Cross-compiling improvements

[the-horo]

Cross-compiling the ldc2 runtime is a relatively painless process, either by using ldc-build-runtime or calling cmake in the runtime subdirectory. However, making the produced binaries usable requires manual intervention, like having to add a new section to ldc2.conf.

I want to work on improving this so I want to discuss possible changes and goals to know if I should implement them or not. Feel free to propose modifications or other features you want since I don't mind working on it, I just want to know what I'm striving for.

Goals

• cmake -S runtime -D... should be enough to cross-compile the runtime, if a cross-toolchain is present
• The runtime should be able to be cross-compiled, and then programs could be built against it
• The runtime should be able to be cross-compiled, and then installed on another system. (this may require a different cmake configuration compared to the above item)
• Cross-compiling the runtime should produce the same files as building it natively
• The compiler should be able to be cross-compiled and installed, without tricks like ninja bin/ldc2 bin/timetrace2txt...
• ninja install should be sufficient to install a cross-compiled runtime
• ldc-build-runtime may be extended to set up things that are needed for cross compiling (like downloading a cross C toolchain) but aren't part of the ldc2 project.

Possible implementations

Isolate the runtime cmake from the compiler cmake more

For cross-compiling to work fully all options that affect both the runtime and the compiler should be specified twice, once in the runtime cmake and another time in the compiler cmake. This also means that the cmake files can no longer rely on options being transitively set (like the runtime assuming that llvm is present or the compiler assuming that a ldc2.conf file has been generated)

This should be able to be accomplished without too much code duplication but with slight complexity by, for example, introducing a LdcCommon module that contains all the build settings that affect both the compiler and the runtime.

Another pain point is ldc2.conf. I don't see a way that it can stay as is, there's no way to cleanly have both the compiler and runtime append their settings to it, unless it is split across multiple files. For example, a compiler could generate a wasm.conf file that contains the default switches for building for wasm and the runtime could generate x64.conf that contains switches, libdirs and rpath for building for x64. I detail this in the next section.

In terms of implementation, there should be one added cmake setting, that which controls skipping going into the runtime subproject if cmake is invoked for the compiler. Additionally some variables that affect both the compiler as well as the runtime build would need to be exposed to users (if they aren't already) so that the projects could be built independently.

The installed ldc2.conf must become a directory

There's no way for ninja install to install (without overwriting) config files from different builds unless the paths differ.

This is a breaking change but it can be alleviated by:

1. Introducing a deprecation period in which users are warned that the config file will change to a directory

2. Supporting both a file and a directory config, but cross-compiling will only work (without intervention) if the user has a directory config.

The first option is the lazy one, with minimal code footprint. The second require a little bit more code but the ldc-build-runtime could be used to install the config file, by concatenating the build generated file to the ldc2.conf present on the user system.

This choice is solely for how ldc2.conf is installed, the compiler would still support loading both a file or a directory config, if the user points it to it.

I prefer the second option since it doesn't really have a downside is pretty minimal but the UX is better.

On the details of implementation it would require two cmake variables (names may be changed)

• LDC_CONFIG_PREFER_DIR - install the config files as a directory instead of as a file (default disabled for backwards compatibility)
• LDC_DONT_INSTALL_CONFIG - skip installing the config file.
  This is probably the easiest way to avoid overwriting a ldc2.conf file installed on a user system. ldc-build-runtime would set this variable and pick up the config file from the build directory in this case.

Make the section name in the runtime ldc2.conf configurable.

There are two reasons for this:

• when cross-compiling having a section titled default would be broken
• picking another option like the default triple (similar to -DMULTILIB=ON) may be unwanted since the cross-compiled objects may be used for more targets compared to the target triple (or the same target may have more spellings like aarch64-windows and arm64-windows).

This would add yet another cmake option. The default value for this setting can stay the target triple (without the vendor, like x86_64-.*-linux-gnu). This is the same way MULTILIB handles it.

Support for target-dependent headers

This is more of a future-proof thing, but imagine you want to target some linux system and use functions provided by glibc. If you are targeting glibc-2.28 it is expected that you won't have the same functions available as on glibc-2.40. Of course, core.sys.linux.* doens't currently have that capability but I think that having support for this will pay off in the future.

The implementation would be simple. The runtime would install two config files:

1. runtime-common.conf

   default: {
       post-switches = [ "-I%%ldcbinarypath%%/../include/d" ]
   }
   

2. runtime-lib<suffix>.conf

   <triple>: {
       post-switches ~= [ "<target_specific_switch>" ]
       lib-dirs = [ "%%ldcbinarypath%%/../lib<suffix>" ]
   }
   

Then, the same build would install platform-independent headers (like phobos, most of druntime) into include/d and the rest of the headers into whichever directory it deems appropriate (one example could be include-lib<suffix>).

Since there are currently no platform-specific headers there's no reason to consider which target specific switch would be present or where to install the headers. The only relevant thing is having the split between the two config files.

When another runtime would be built and installed it would overwrite the runtime-common.conf file, but since it contains the same include path there would be no issue, as well as overwriting all of include/d but, again, the files would stay the same so it will be all right.

Remove (or deprecate) MULTILIB

If cross-compiling becomes usable out-of-the-box -DMULTILIB=ON could be replaced by calling ldc-build-runtime --dFlags=-m32. Additionally, since cmake doesn't support looking for packages for multiple architectures in the same build, the -DMULTILIB=ON setting would prevent the x86 build from either finding zlib or llvm, whereas ldc-build-runtime would be able to find them.

-DMULTILIB also affects the compiler by running the dmd testsuite for -m32 as well. In order to avoid losing this feature a cmake option could be introduced which will contain a list of triple-like flags to be tested.

Standard way of specifying the triple

Since the main topic is cross-compiling the runtime, a very important thing is being able to specify the target system. Currently this is supported by using D_FLAGS for the runtime and D_COMPILER_FLAGS for the compiler. This is different from the norm as those types of settings are usually for flags that aren't vital for the build (like -O and -g). The standard way of specifying the cross-compiler is by putting its flags into the compiler variable. Cmake does this by having -DCMAKE_C_COMPILER be set to a list of two elements, firstly the compiler path followed by a space-separated list of arguments. For example -CMAKE_C_COMPILER='D:/llvm 19/bin/clang;-target i686-pc-linux-gnu'.

In ldc2, D_COMPILER can be used in a similar way, albeit it always splits by (unqouted) spaces. So -DD_COMPILER='"C:/ldc 1.40/bin/ldc2" -mtriple i686-pc-linux-gnu' would be the equivalent. Unfortunately LDC_EXE_FULL doesn't have such capabilities and the only way to pass the triple is through D_FLAGS.

For the sake of standardisation I propose (again) that LDC_EXE_FULL either follow the CMAKE_C_COMPILER or D_COMPILER format. Having it follow CMAKE_C_COMPILER is slightly preferred since that won't require users to add quotes if their full path contained spaces before.

ldc-build-runtime being able to setup a cross-C or other toolchains

This is pretty low on my list of priorities but I will try to get to it after the other issues are solved so I don't have many comments (nor do I currently have the knowledge on how non-Gentoo systems handle their cross-toolchains).

[kinke]

I'll read the wall of text another time. :)

I'm interested in your specific use case. Wrt. Fedora, is this mainly just for the 'manual' Linux multilib package, or do you plan on building & providing more cross-compiled runtime libs directly?

For the official packages, I found it quite okay to build via ldc-build-runtime and patch the ldc2.conf manually, with knowledge about the whole package/config and available targets. E.g., with your ~= array-appending feature, the ldc2.conf for the universal macOS package (supporting 4 Apple platforms, and wasm to the current preliminary degree) could look like this:

default:
{
    switches = [
        "-defaultlib=phobos2-ldc,druntime-ldc",
    ];
    post-switches = [
        "-I%%ldcbinarypath%%/../import",
    ];
    lib-dirs = [];
    rpath = "";
};

"x86_64-apple-":
{
    lib-dirs = [
        "%%ldcbinarypath%%/../lib-x86_64",
    ];
    rpath = "%%ldcbinarypath%%/../lib-x86_64";
};

"arm64-apple-":
{
    lib-dirs = [
        "%%ldcbinarypath%%/../lib-arm64",
    ];
    rpath = "%%ldcbinarypath%%/../lib-arm64";
};

"x86_64-apple-ios":
{
    switches ~= [
        "-Xcc=-isysroot",
        "-Xcc=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk",
    ];
    lib-dirs = [
        "%%ldcbinarypath%%/../lib-ios-x86_64",
    ];
    rpath = "%%ldcbinarypath%%/../lib-ios-x86_64";
};

"arm64-apple-ios":
{
    switches ~= [
        "-Xcc=-isysroot",
        "-Xcc=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk",
    ];
    lib-dirs = [
        "%%ldcbinarypath%%/../lib-ios-arm64",
    ];
    rpath = "%%ldcbinarypath%%/../lib-ios-arm64";
};

"^wasm(32|64)-":
{
    switches = [
        "-defaultlib=",
        "-L-z", "-Lstack-size=1048576",
        "-L--stack-first",
        "-link-internally",
        "-L--export-dynamic",
    ];
    lib-dirs = [];
};

This is currently done in

ldc/.github/actions/merge-macos/action.yml

Lines 13 to 90 in c26e843

	- name: Extract & merge artifacts
	shell: bash
	run: |
	set -euxo pipefail
	mkdir ldc2-{x86_64,arm64}
	tar -xf artifacts/ldc2-*-x86_64.tar.xz --strip 1 -C ldc2-x86_64
	tar -xf artifacts/ldc2-*-arm64.tar.xz --strip 1 -C ldc2-arm64
	cp -R ldc2-x86_64 ldc2-universal
	cd ldc2-universal
	# rename/copy lib dirs
	mv lib lib-x86_64
	cp -R ../ldc2-arm64/lib lib-arm64
	cp -R ../ldc2-arm64/lib-ios-arm64 ./
	# merge executables to universal ones
	for exe in bin/*; do
	rm $exe
	lipo -create -output $exe ../ldc2-x86_64/$exe ../ldc2-arm64/$exe
	done
	# merge LTO plugins to universal one (and keep in hardcoded lib/)
	mkdir lib
	lipo -create -output lib/libLTO.dylib lib-x86_64/libLTO.dylib lib-arm64/libLTO.dylib
	rm lib-{x86_64,arm64}/libLTO.dylib
	# ldc2.conf: replace the default section and add extra sections
	# note: x86_64-apple-ios section already exists
	sections="
	default:
	{
	// default switches injected before all explicit command-line switches
	switches = [
	\"-defaultlib=phobos2-ldc,druntime-ldc\",
	];
	// default switches appended after all explicit command-line switches
	post-switches = [
	\"-I%%ldcbinarypath%%/../import\",
	];
	// default directories to be searched for libraries when linking
	lib-dirs = [];
	// default rpath when linking against the shared default libs
	rpath = \"\";
	};
	\"x86_64-apple-\":
	{
	lib-dirs = [
	\"%%ldcbinarypath%%/../lib-x86_64\",
	];
	rpath = \"%%ldcbinarypath%%/../lib-x86_64\";
	};
	\"arm64-apple-\":
	{
	lib-dirs = [
	\"%%ldcbinarypath%%/../lib-arm64\",
	];
	rpath = \"%%ldcbinarypath%%/../lib-arm64\";
	};
	\"arm64-apple-ios\":
	{
	switches = [
	\"-defaultlib=phobos2-ldc,druntime-ldc\",
	\"-Xcc=-isysroot\",
	\"-Xcc=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk\",
	];
	lib-dirs = [
	\"%%ldcbinarypath%%/../lib-ios-arm64\",
	];
	rpath = \"%%ldcbinarypath%%/../lib-ios-arm64\";
	};"
	perl -0777 -pi -e "s|\\ndefault:\\n.+?\\n\\};|$sections|s" etc/ldc2.conf
	cat etc/ldc2.conf

The 2 x86_64 and arm64 source packages already contain their macOS+iOS libs, so it's just a matter of copying the lib dirs and tweaking the .conf. No building/CMake involved for this merge-to-universal-package step.

Edit: And building, copying & .conf-extending the iOS libs in the source packages goes like this:

ldc/.github/actions/5a-ios/action.yml

Lines 26 to 52 in c26e843

	bootstrap-ldc/bin/ldc-build-runtime --ninja \
	--buildDir=build-libs-ios \
	--dFlags="-mtriple=$triple" \
	--ldcSrcDir="$PWD/ldc" \
	CMAKE_SYSTEM_NAME=iOS \
	CMAKE_OSX_SYSROOT="$sysroot" \
	CMAKE_OSX_ARCHITECTURES="$arch" \
	CMAKE_OSX_DEPLOYMENT_TARGET="$deployment_target" \
	BUILD_LTO_LIBS=ON
	mkdir "installed/lib-ios-$arch"
	cp -a build-libs-ios/lib/*.{a,dylib,o} "installed/lib-ios-$arch/"
	section="
	\"$arch-apple-ios\":
	{
	switches = [
	\"-defaultlib=phobos2-ldc,druntime-ldc\",
	\"-Xcc=-isysroot\",
	\"-Xcc=$sysroot\",
	];
	lib-dirs = [
	\"%%ldcbinarypath%%/../lib-ios-$arch\",
	];
	rpath = \"%%ldcbinarypath%%/../lib-ios-$arch\";
	};"
	echo "$section" >> installed/etc/ldc2.conf

[kinke]

Cross-compiling the runtime should produce the same files as building it natively

OTOH, that should be the case, more or less, with the notable jit-rt (for dynamic-compile) exception, as that requires the LLVM libs for that target. Edit: Oh, and all the compiler-rt libs are missing as well (incl. builtins, profile, sanitizers...), as we'd need those from a target-compatible LLVM build too.

The compiler should be able to be cross-compiled and installed, without tricks like ninja bin/ldc2 bin/timetrace2txt...

IIRC, that trick is only employed to restrict the cross-build to the executables, excluding druntime and Phobos, which have been cross-compiled earlier (via ldc-build-runtime), so that the executables can be linked against those. The cross-compiled compiler can normally not be run on the host, so cannot build druntime and Phobos.

But fully agreed, needing a list of all executables is ugly, so maybe some CMake special casing for CMAKE_CROSSCOMPILING wouldn't be too painful.

A major hurdle for cross-compiling LDC itself is the need for target-compatible LLVM libs, coupled with a host-runnable llvm-config executable/script. The latter is surely the biggest problem. Maybe all the stuff we need would nowadays be derivable from the LLVM-bundled CMake files.

ninja install should be sufficient to install a cross-compiled runtime

You mean to avoid something like cp -a build-libs-ios/lib/*.{a,dylib,o} "installed/lib-ios-$arch/"? That'd be nice, yes, given some install dir to copy/install to. (The main difference to the build dir is the lack of some unneeded build artifacts.)

[kinke]

Remove (or deprecate) MULTILIB

I guess that would lead to a massive CMake simplification, so I'm in favor of removing it soonish.

[kinke]

Standard way of specifying the triple

For the official packages, I handle this via the DFLAGS env variable, which is respected by the ldmd2 (not ldc2!) host compiler. For macOS:

ldc/.github/actions/3-build-cross/action.yml

Line 83 in c26e843

echo "DFLAGS=-mtriple=$triple -L-L$PWD/build-cross-libs/lib -Xcc=-target -Xcc=$triple -Xcc=-isysroot -Xcc=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" >> $GITHUB_ENV

and Android:

ldc/.github/actions/3-build-cross/action.yml

Line 118 in c26e843

echo "DFLAGS=-mtriple=$triple -L-L$PWD/build-cross-libs/lib -gcc=$PWD/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/$triple-clang" >> $GITHUB_ENV

[the-horo]

I'm interested in your specific use case. Wrt. Fedora, is this mainly just for the 'manual' Linux multilib package, or do you plan on building & providing more cross-compiled runtime libs directly?

I was planning to support cross-compiling for Gentoo packages (which users build themselves) and i did get it working to some degree but it is hacky so I want to implement it cleanly here. And if I'm going to do that I might as well cross out some other fixes/improvements.

For the official packages, I found it quite okay to build via ldc-build-runtime and patch the ldc2.conf manually, with knowledge about the whole package/config and available targets.

Modifying already installed files is pretty much the only thing I can't do so that's why I insist so much on having ldc2.conf being a directory. But to be really honest, on Gentoo I wouldn't benefit from these changes at all. Let me explain

To get a cross-gcc on Gentoo you would need to install cross-aarch64-unknown-linux-gnu/gcc in /. Then you use it to cross-compile other packages. If the package has any dependencies you would need to compile them as well, however, their are just normal packages with the sole distinction that they are built with aarch64-unknwon-linux-gnu-gcc and installed in a separte directory, which usually is /usr/aarch64-unkonwn-linux-gnu.

So, based on the same logic I could install cross-aarch64-unknown-linux-gnu/ldc2-runtime and, assuming I implemented what I wanted here it would work. The thing is that I don't need that because packages already depend on ldc2-runtime which would be installed in /usr/aarch64-unknown-linux-gnu. So all I need to do is simply generate a ldc2.conf that points its libdir to /usr/aarch64-unknown-linux-gnu/..../ldc2/lib. Very similar to what is done in CI except ldc2.conf is not installed.

A major hurdle for cross-compiling LDC itself is the need for target-compatible LLVM libs, coupled with a host-runnable llvm-config executable/script.

Another reason for me doing cross-compiling in Gentoo is the addition of a way to generate a llvm-config shell script for cross-compiling, again similar to what is done in CI.

You mean to avoid something like cp -a build-libs-ios/lib/*.{a,dylib,o} "installed/lib-ios-$arch/"? That'd be nice, yes, given some install dir to copy/install to. (The main difference to the build dir is the lack of some unneeded build artifacts.)

On Gentoo specifically cross-compiling means building a package for another system, that will be installed on that system. So the package needs to contain the same files that it does when built natively. This means that I need the same dir structure, the import files, (possible) the config file so it would be easier if ninja install simply installed what is needed.

The only thing that I would benefit from all of these is ldc2.conf being able to be a directory since that helps with the manual linux multilib because I can't have simultaneous access to the ldc2.conf generated by the compiler and the ldc2.conf generated by the runtime since those are two distinct builds and by the time one is installed it's too late to modify it. Currently I'm using the one from the runtime since it contains everything that is needed with the exception of the compiler-rt libdir.

---

All in all, I can get away with having some code on the Gentoo side handling these small issues and it would be OK. But it's enough code to make me want to change something upstream. Because I want that, instead of doing what is minimally sufficient for me I'm trying to get the stuff I touch in the best state that it can be. To do this I sat down and tried to figure out what would be the ideal way that this code would act, how it would be most intuitive/standard, have the least bugs, and bring the best user experience. so I can up with this. I don't know if all of these are achievable but I will try to get as close as possible.

[kinke]

So, based on the same logic I could install cross-aarch64-unknown-linux-gnu/ldc2-runtime and, assuming I implemented what I wanted here it would work. The thing is that I don't need that because packages already depend on ldc2-runtime which would be installed in /usr/aarch64-unknown-linux-gnu. So all I need to do is simply generate a ldc2.conf that points its libdir to /usr/aarch64-unknown-linux-gnu/..../ldc2/lib.

Would this extra ldc2.conf file be 'installed' as part of the cross ldc2-runtime package directly? IIUC, this would automatically make an optionally installed native Gentoo LDC package able to target that target.

Thinking about how this could potentially work with what we currently have, without having to patch the compiler's ldc2.conf: how about a aarch64-linux-gnu-ldc2 package, depending on (native) ldc2 and ldc2-runtime-dev:aarch64 (or however you call the cross-runtime package) [edit: oh and probably also on aarch64-linux-gnu-gcc for cross-linking], and just additionally creating a suited ldc2.conf somewhere, as well as a aarch64-linux-gnu-ldc2 script forwarding to the native ldc2, with hardcoded switches -conf=…/ldc2.conf -mtriple=aarch64-linux-gnu? Edit: This would be similar to what the Android NDK uses.

[the-horo]

Would this extra ldc2.conf file be 'installed' as part of the cross ldc2-runtime package directly? IIUC, this would automatically make an optionally installed native Gentoo LDC package able to target that target.

That would the plan as that would allow ldc2 -mtriple aarch64-unknown-linux-gnu to just work, without introducing another wrapper.

But, it would have consequences.

Having a cross-aarch64-unknown-linux-gnu/ldc2-runtime package

pros

• works when invoked by the user as a standard ldc2 -mtriple aarch64-unknown-linux-gnu or aarch64-unknown-linux-gnu-ldc2

cons

• requires manual intervention from the user for creating the package, as I can't create the package for every possible triple
  This would involve:

  ln -s ../dev-libs/ldc2-runtime cross-aarch64-unknown-linux-gnu/ldc2-runtime
  

  As well as having users keep track of the symlink.

• requires manual intervention from the user for installing the package
  The gentoo package manager doesn't allow making dependencies conditional, so I can't say "when cross compiling for triple depend on cross-<triple>/ldc2-runtime". So the users have to install it themselves.

• it would make the runtime be built twice when installing packages from the gentoo repos
  Again, since I can't make the dependencies conditional all I can tell the build system is:

  • depend on dev-lang/ldc2 (the compiler) installed in / (so natively built)
  • depend on dev-libs/ldc2-runtime installed for the target system (which means / for native builds or /usr/aarch64-unknown-linux-gnu for cross)
  • depend on dev-libs/ldc2-runtime at runtime (so installed in / on another system)

  This would make it so that every package built with ldc2 would require dev-lang/ldc2, dev-libs/ldc2-runtime, and, cross-aarch64-unknown-linux-gnu/ldc2-runtime

Having dev-libs/ldc2-runtime installed under /usr/aarch64-unknown-linux-gnu

pros

• doesn't require user intervention when building packages with the gentoo package manager
  This is because emerge knows to pull dev-libs/ldc2-runtime in /usr/aarch64-unknown-linux-gnu as part of the dependency graph.

cons

• requires the user to change ldc2.conf if they want to manually cross-compile
  They would probably have to do it anyway since since the gentoo-shipped configuration would be undesired, as it does things like embedding -link-defaultlib-shared, which is fine when you have a package manager that would make sure those libraries are available but not so fine when you're manually building and installing stuff.

---

Given that both options involve some level of intervention from the user I would rather go for the second since:

• it's less interaction
• it better supports by goal, which is making the Gentoo build system (not users directly) be able to cross-compile D packages
• requires adding less edge-cases

---

As a side-note, the whole point why cross-aarch64-unknown-linux-gnu-gcc is needed is because gcc needs to be compiled for each triple you want to support, it's not like clang or ldc2 where you can just pass -target or -mtriple, aarch64-unknown-linux-gnu-gcc is a completely different executable.