richiejp logo

Minimal Linux VM cross compiled with Clang and Zig

One of my jobs is to write and run Linux kernel tests. These are part of the Linux Test Project (LTP). Building and running kernel tests requires a fair amount of unusual tooling.

At least if you are concerned about more than one platform and kernel version. Writing a test and making sure it builds and runs on all architectures and Linux distros is a pain.

It would be a big achievement just to be able to quickly and simply run a new test on all the architectures QEMU and Linux jointly support. It would also be nice to be able to test a kernel patch in a reasonable time.

Of course there are various existing solutions to this. One is to use a build farm that someone has setup. If you pick a major distro, you can create a package and have their build farm compile it across multiple arches.

You could then run the tests in something like OpenQA… Ah no, hold on. I said quickly and simply. In that case, the closest solution I have found is buildroot.

Buildroot compiles GCC and everything else from source. It’s good at cross compiling and supports LTP. There is still a lot of stuff going on here though which we don’t want or care about.

In general there is a lot of profit to be had in stripping back unnecessary layers. The problem is it has a high up front cost. Meanwhile complexity is cheap as Halvar Flake says.

Anyway we don’t even need busybox (for most tests anyway, some require sh). We are developing a test executor (LTX) which can run as init. All we need is a VM with LTX and some tests inside.

Update2: I created a small project from this where a Zig program is running as init

Update: Andrea Cervesato rewrote LTX again (I wrote it twice, hopefully third time is a charm) and I added Zig cross compilation to that version too.

LTX now works with Kirk and I renewed my plan to automate compiling Linux and direct boot VMs with LTX as init. As of writing this is still very much work in progress.

QEMU allows us to boot directly from a kernel image and an initrd file. We don’t need GRUB installed in a disk image. The initrd can contain LTX and some LTP tests.

So all we need to do is cross compile the kernel, cross compile LTX and some LTP tests. Then create an initrd file. For this I used my favorite new shiny thing which I am itching for an excuse to use.

Building Linux

Clang LLVM supports cross compilation out of the box. This doesn’t work well with userland due to a dependency on libgcc (see below). However Linux doesn’t require this. Using Clang with Linux is as simple as adding LLVM=1 to the Make command.

$ cd $linux
$ make LLVM=1 ARCH=arm64 defconfig
$ make LLVM=1 ARCH=arm64 menuconfig
$ make LLVM=1 ARCH=arm64 -j$(nproc)
$ cp arch/arm64/boot/Image.gz $ltx/cross/aarch64/

GCC seems to need compiling for each architecture’s backend. I suppose that one should be able to use their distro’s cross compiler GCC package. However I haven’t had much luck with it.

Note that I wasn’t able to get zig cc to work with the kernel. It’s not clear if it’s worth the effort either. At least not until Zig implements its own C compiler instead of wrapping Clang. And yes someone is working on that.

Building Userland

Userland is complicated by libc’s dependency on the compiler runtime library (compiler-rt, libgcc). LLVM has it’s own compiler-rt, but it is missing features that are implemnted by libgcc. Cross compiling GCC is a farce, hence why we are using Clang in the first instance.

Luckily the Zig language bundles Clang, its own compiler-rt and some libc’s (e..g musl, glibc). Zig’s compiler-rt appears to be incomplete as well, however it doesn’t seem to matter for our purposes.

It appears that Zig compiles only the parts of libc required for our application from source. This is only a small subset of musl which doesn’t include some floating point functionality which is missing from LLVM’s and Zig’s runtime libraries.

Also even though the LTX executable produced by Zig is statically compiled. It is relatively small at 126K. This is double the size compared to being dynamically linked to musl. However we can live with this.

LTX

So LTX can be compiled by Zig as follows.

$ cd $runltp-ng/ltx
$ zig cc --target=aarch64-linux-musl -o ltx ltx.c
$ cp ltx cross/initrd/init

We could of course put this in a Makefile or build.zig.

LTP

My first attempt at cross compiling LTP with Zig has not been entirely successful. However it appears that it can compile most tests. This is good enough for now and also a pleasant surprise.

$ cd $ltp
$ make autotools
$ ./configure --prefix=(realpath ../ltp-install/) CC='zig cc --target=aarch64-linux-musl' --host=aarch64
$ make -j$(nproc)

Test executables can be copy and pasted into $ltx/cross/initrd/bin or similar. Unfortunately the test executables come out as ~600-700K. This is a potential problem for LTP, because it has thousands of them.

This is also LTP’s fault though, at least to some extent. The LTP library is built first and rolled up into an archive. Then it is linked into each test executable.

It would be better for Zig if all the library sources were passed in each time. This should allow much better dead code elimination. Possibly it would improve compile times as well. Zig is designed to compile from source.

initrd

Unless we embed init (LTX) inside the kernel image. We need an initial system image which the kernel can load init from. This can be created with cpio.

$ cd $ltx/cross/initrd
$ find . | cpio -H newc -o | gzip -n > ../aarch64/initrd.cpio.gz

Thus we have a compressed system image just with /init (LTX) in the root directory.

N.B. LTX must be located at /init (assuming default kernel config). If we don’t do that then Linux starts trying to use some deprecated boot procedure. This then results in a confusing error message about not being able to find the root partition or whatever.

Running

The kernel and initramfs can be direct booted by QEMU

$ qemu-system-aarch64 -m 1G \
                      -smp 2 \
                      -display none \
                      -kernel aarch64/Image \
                      -initrd aarch64/initrd.cpio.gz \
                      -machine virt -cpu cortex-a57 \
                      -serial stdio \
                      -append 'console=ttyAMA0 earlyprintk=ttyAMA0'

LTX starts up fine as init, the last lines from the kernel log and LTX’s stderr are below.

[    1.278618] Run /init as init process
[ltx.c:main:1075] Running as init

Next we need to communicate with LTX, using serial or whatever, and get it to run some tests. I’ll leave that for another day.