Cross Compilation Is Pain

August 19, 2023

I got my ROCKPro64 a couple months ago as a birthday present from my wonderful girlfriend. My job kept me busy, so the device had to just stay on my shelf for a while. But now that I’m on vacation, I finally have the time to play with it.

As a typical Gentoo enjoyer, the first thing I’m going to do is install Gentoo on it. I’ve had some success building Das U-Boot and the Linux kernel, so now all that’s left is the root filesystem.

I could take Gentoo’s stage3 tarball for arm64 and just use that. And I did, but what’s next? The resulting system is too minimal for actual use, and I am interested in getting my regular setup on this device. I try to stick to “minimalist” software like i3, st, and so on, but even so one can’t avoid huge packages like gcc, LLVM, and firefox. This little ARM computer can build these too, but it’s pretty obvious that its slower CPU and a cheap SD card I got with some other device can’t match my desktop’s Ryzen CPU and an NVME SSD. One could use binary packages, but where’s the fun in that? After all, in this game, you gain points for using software you built from source code.

There are several ways to increase your score:

• Just accept long build times and build everything on native hardware. I once built libreoffice on my Celeron netbook. The process took about 12 hours, so something tell me it’s going to take more than a full day on ROCKPro64. Not that I need libreoffice on this machine, but building the big lads like LLVM and firefox may take a bit of time.
• Install an ARM virtual machine on your desktop and use it to build everything. It’s a pretty straightforward process, but you will pay the price of emulating a different CPU architecture. Funtoo provides a nice little utility called fchroot if you’re interested in going that route.
• Use a cross-compiler. This allows the build process to be as efficient as it can be, but at the cost of increased complexity. The build system used by the package you’re trying to build needs to be aware that it’s running in a “foreign” environment and adapt as needed. This makes some checks impossible (or at least complicated): you can’t run the code you build directly, which is what a lot of “configure” scripts rely upon. Because of that sometimes you end up overriding these checks using your knowledge of the target system.

As you may have guessed, I’m going to stick to the last option. After all, I’ve got a cross-compiler installed already, I used it to build Das U-Boot and the Linux kernel. Moreover, Gentoo has this cool feature called crossdev, which allows you to cross-build pretty much anything you have in your portage tree.

Preparation

After installing crossdev and the aarch64-unknown-linux-gnu toolchain, we get a directory that you can treat like you would a rootfs of our ARM system: /usr/aarch64-unknown-linux-gnu/.

Since we’re going to use portage to build and install our packages there, we might as well configure it properly. So go make your favourite adjustments to the etc/make.conf file in there and don’t forget to throw in your favorite USE flags.

These crossdev environments actually use the embedded portage profile, which may not be what you want. I noticed that the stage3 tarball for arm64 uses the default/linux/arm64/17.0 profile, so decided to stich with that. I’m not sure if you can use eselect in these environments, so I changed the make.profile symlink in etc/portage manually.

Let’s build the packages

Once it’s all set up, you should be able to start building your packages. I decided to be a little sly and just added the packages of interest to the world set, which in my case was in /usr/aarch64-unknown-linux-gnu/var/lib/portage/world.

Now we’re ready to build everything using this simple command:

# aarch64-unknown-linux-gnu-emerge -uDNav @world

If you’re lucky, you won’t encounter any problems and all your packages will be installed successfully.

Of course, I wasn’t so lucky. There was a small series of “misfortunes”, which I managed to work around. Some of them I even managed to solve “properly” and hopefully I’ll be able to upstream the fixes soon enough.

Misfortune 1: cairo

Cairo’s build process really wanted to build and run some code for some check. Using the information from here, I simply updated the meson.eclass file to include the property cairo was interested in. There should be a better way to solve that, maybe I’ll take a proper look later.

Misfortune 2: gobject-introspection

The package refused to build because something something introspection and cross-compilation bad. See here.

I just disabled the introspection USE flag for now, hopefully it’s not going to be a problem.

Misfortune 3: dev-perl/JSON-XS-4.30.0

>>> Source configured.
>>> Compiling source in /usr/aarch64-unknown-linux-gnu/tmp/portage/dev-perl/JSON-XS-4.30.0/work/JSON-XS-4.03 ...
make -j24 OTHERLDFLAGS= 'OPTIMIZE=-O2 -pipe -fomit-frame-pointer -march=armv8-a -mtune=cortex-a72.cortex-a53'
Running Mkbootstrap for XS ()
chmod 644 "XS.bs"
"/usr/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- XS.bs blib/arch/auto/JSON/XS/XS.bs 644
cp XS/Boolean.pm blib/lib/JSON/XS/Boolean.pm
cp XS.pm blib/lib/JSON/XS.pm
"/usr/bin/perl" "/usr/lib64/perl5/5.38/ExtUtils/xsubpp"  -typemap '/usr/lib64/perl5/5.38/ExtUtils/typemap' -typemap '/usr/aarch64-unknown-linux-gnu/tmp/portage/dev-perl/JSON-XS-4.30.0/work/JSON-XS-4.03/typemap'  XS.xs > XS.xsc
mv XS.xsc XS.c
x86_64-pc-linux-gnu-gcc -c   -O2 -pipe -march=znver2 -fno-strict-aliasing -DNO_PERL_RAND_SEED -fwrapv -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -pipe -fomit-frame-pointer -march=armv8-a -mtune=cortex-a72.cortex-a53   -DVERSION=\"4.03\"
 -DXS_VERSION=\"4.03\" -fPIC "-I/usr/lib64/perl5/5.38/x86_64-linux/CORE"   XS.c
cc1: error: bad value ‘armv8-a’ for ‘-march=’ switch
cc1: note: valid arguments to ‘-march=’ switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client rocketlake icelake-server casc
adelake tigerlake cooperlake sapphirerapids emeraldrapids alderlake raptorlake meteorlake graniterapids bonnell atom silvermont slm goldmont goldmont-plus tremont sierraforest grandridge knl knm x86-64 x86-64-v2 x86-64-v3 x86-64-v4 eden-x2
nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 lujiazui k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 znver3 znver4 btver1 btver2 native
cc1: error: bad value ‘cortex-a72.cortex-a53’ for ‘-mtune=’ switch
cc1: note: valid arguments to ‘-mtune=’ switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client rocketlake icelake-server casc
adelake tigerlake cooperlake sapphirerapids emeraldrapids alderlake raptorlake meteorlake graniterapids bonnell atom silvermont slm goldmont goldmont-plus tremont sierraforest grandridge knl knm intel x86-64 eden-x2 nano nano-1000 nano-2000
 nano-3000 nano-x2 eden-x4 nano-x4 lujiazui k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 znver3 znver4 btver1 btver2 generic native
make: *** [Makefile:334: XS.o] Error 1

Hmmm. It’s trying to use my regular x86_64 compiler to compile and link the ARM binaries.

To fix this, I updated perl-module.eclass to check CBUILD/CHOST and add the necessary CC and LD arguments to Makefile.PL. I’ve no idea if a similar change need to be made for packages that use Build.PL, time will tell.

Misfortune 4: wpa_supplicant

 * Building wpa_supplicant
make -j24 V=1 BINDIR=/usr/sbin
Package libnl-3.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libnl-3.0.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libnl-3.0', required by 'virtual:world', not found

... snip snip ...

../src/drivers/driver_nl80211_capa.c:12:10: fatal error: netlink/genl/genl.h: No such file or directory
   12 | #include <netlink/genl/genl.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

The compiler can’t find the header file installed by the linbl package. And the configuration script doesn’t see the pkg-config file installed by the same package. libnl is certainly installed in my ARM environment, so what’s up with that?

# ls /usr/aarch64-unknown-linux-gnu/usr/lib*/pkgconfig/
/usr/aarch64-unknown-linux-gnu/usr/lib/pkgconfig/:
SPIRV-Tools-shared.pc        cairo-xlib.pc    gmodule-2.0.pc
SPIRV-Tools.pc               cairo.pc         gmodule-export-2.0.pc
alsa-topology.pc             dav1d.pc         gmodule-no-export-2.0.pc
alsa.pc                      dbus-1.pc        gmp.pc
cairo-fc.pc                  egl.pc           gmpxx.pc
cairo-ft.pc                  fontconfig.pc    gnutls.pc
cairo-gobject.pc             fontenc.pc       gobject-2.0.pc
cairo-pdf.pc                 fontutil.pc      gpgme-glib.pc
cairo-png.pc                 freetype2.pc     gpgme.pc
cairo-ps.pc                  fribidi.pc       graphite2.pc
cairo-script-interpreter.pc  gio-2.0.pc       gthread-2.0.pc
cairo-script.pc              gio-unix-2.0.pc  harfbuzz-cairo.pc
cairo-svg.pc                 gl.pc            harfbuzz-subset.pc
cairo-tee.pc                 glesv1_cm.pc     harfbuzz.pc
cairo-xcb-shm.pc             glesv2.pc        ice.pc
cairo-xcb.pc                 glib-2.0.pc      ksba.pc
cairo-xlib-xrender.pc        glx.pc           libass.pc

/usr/aarch64-unknown-linux-gnu/usr/lib64/pkgconfig/:
blkid.pc   formt.pc      hogweed.pc  libcrypt.pc  libidn2.pc   libmagic.pc
expat.pc   formtw.pc     libacl.pc   libcurl.pc   libip4tc.pc  libmnl.pc
expatw.pc  formw.pc      libattr.pc  libdw.pc     libip6tc.pc  libpcre2-16.pc
fdisk.pc   gpg-error.pc  libb2.pc    libelf.pc    libiptc.pc   libpcre2-8.pc
form.pc    history.pc    libcap.pc   libevdev.pc  liblzma.pc   libpcre2-posix.pc

Oh, so there are two different pkg-config directories in that environment, and aarch64-unknown-linux-gnu-pkg-config is not aware of that. Sound like a bug, but let’s implement a small workaround to change that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

diff --git a/aarch64-unknown-linux-gnu-pkg-config b/aarch64-unknown-linux-gnu-pkg-config
index f178147..677c2bd
--- a/aarch64-unknown-linux-gnu-pkg-config
+++ b/aarch64-unknown-linux-gnu-pkg-config
@@ -123,6 +123,7 @@ export PKG_CONFIG_SYSTEM_LIBRARY_PATH="${PREFIX}/usr/${libdir}:${PREFIX}/${libdi
 # Set the pkg-config search paths to our staging directory.
 #
 PKG_CONFIG_LIBDIR="${PKG_CONFIG_LIBDIR}${PKG_CONFIG_LIBDIR:+:}${PKG_CONFIG_ESYSROOT_DIR}/usr/${libdir}/pkgconfig:${PKG_CONFIG_ESYSROOT_DIR}/usr/share/pkgconfig"
+PKG_CONFIG_LIBDIR="${PKG_CONFIG_LIBDIR}:${PKG_CONFIG_ESYSROOT_DIR}/usr/lib/pkgconfig"

 #
 # Sanity check the output to catch common errors that do not

Misfortune 5: vim

The configuration process failed:

checking for shmat... yes
checking for IceConnectionNumber in -lICE... yes
checking if X11 header files can be found... yes
checking for _XdmcpAuthDoIt in -lXdmcp... yes
checking for IceOpenConnection in -lICE... yes
checking for XpmCreatePixmapFromData in -lXpm... yes
checking if X11 header files implicitly declare return values... no
checking size of wchar_t is 2 bytes... configure: error: failed to compile test program

The code it tries to compile looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#include <X11/Xlib.h>
#if STDC_HEADERS
# include <stdlib.h>
# include <stddef.h>
#endif
		int main()
		{
		  if (sizeof(wchar_t) <= 2)
		    exit(1);
		  exit(0);
		}

Compiles just fine, albeit with a bunch of warnings. Let’s look at configure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

if test "$cross_compiling" = yes
then :
  as_fn_error $? "failed to compile test program" "$LINENO" 5
else $as_nop
  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
/* end confdefs.h.  */

#include <X11/Xlib.h>
#if STDC_HEADERS
# include <stdlib.h>
# include <stddef.h>
#endif
		int main()
		{
		  if (sizeof(wchar_t) <= 2)
		    exit(1);
		  exit(0);
		}
_ACEOF
echo "CONFTEST SOURCE"
cat conftest.$ac_ext
if ac_fn_c_try_run "$LINENO"
then :
  ac_cv_small_wchar_t="no"
else $as_nop
  ac_cv_small_wchar_t="yes"
fi

Oh right, we’re trying to run it, but we can’t because it’s for a different CPU architecture. What if we change the code to use a compile-time check instead of a runtime one?

1
2
3
4
5
6
7
8
9

#include <X11/Xlib.h>
#if STDC_HEADERS
# include <stdlib.h>
# include <stddef.h>
#endif
		int main()
		{
		  char arr[sizeof(wchar_t) <= 2 ? 1 : -1];
		}

Builds now.

This looks like a valid fix, maybe I could send it upstream?

And maybe I could, if someone hadn’t done it the day before. But I’m not mad, their solution is much better than mine.

Misfortune 6: mesa

Another configuration failure:

llvm-config found: NO need ['>= 5.0.0']
Run-time dependency LLVM found: NO (tried cmake and config-tool)
Looking for a fallback subproject for the dependency llvm (modules: bitwriter, engine, mcdisassembler, mcjit, core, executionengine, scalaropts, transformutils, instcombine, native)
Building fallback subproject with default_library=shared

../mesa-23.1.4/meson.build:1660:13: ERROR: Neither a subproject directory nor a llvm.wrap file was found.

I’m pretty sure I have llvm-config on my system, why can’t meson find it?

It looks like it’s actually looking for aarch64-unknown-linux-gnu-llvm-config on my main system. And that one is actually absent, perhaps this isn’t implemented by crossdev yet.

We’re in a hurry, so let’s just write a wrapper and place it in /usr/bin/aarch64-unknown-linux-gnu-llvm-config. : )

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/usr/bin/env perl

use strict;
use warnings;

open my $f, '-|', 'llvm-config', @ARGV;

print join ' ', map {
	$_ =~ s/^-L\//-L\/usr\/aarch64-unknown-linux-gnu\//;
	$_ =~ s/^-I\//-I\/usr\/aarch64-unknown-linux-gnu\//;
	$_;
} split /\s+/, join ' ', <$f>;
close $f;

Ahh, much better.

Misfortune 7: librsvg

The build fails with a lot of errors like these:

error[E0425]: cannot find function, tuple struct or tuple variant `Ok` in this scope
    --> /usr/aarch64-unknown-linux-gnu/tmp/portage/gnome-base/librsvg-2.56.3/work/cargo_home/gentoo/smallvec-1.11.0/src/lib.rs:1237:20
     |
1237 |             return Ok(());
     |                    ^^ not found in this scope

error[E0425]: cannot find function, tuple struct or tuple variant `Ok` in this scope
    --> /usr/aarch64-unknown-linux-gnu/tmp/portage/gnome-base/librsvg-2.56.3/work/cargo_home/gentoo/smallvec-1.11.0/src/lib.rs:1257:20
     |
1257 |             return Ok(());
     |                    ^^ not found in this scope

error[E0425]: cannot find function, tuple struct or tuple variant `Err` in this scope
    --> /usr/aarch64-unknown-linux-gnu/tmp/portage/gnome-base/librsvg-2.56.3/work/cargo_home/gentoo/smallvec-1.11.0/src/lib.rs:1495:13
     |
1495 |             Err(self)
     |             ^^^------
     |             |
     |             help: try calling `Err` as a method: `self.Err()`

I guess it’s the first Rust package I’ve encountered. The Rust compiler supports cross-compilation out of the box, so it shouldn’t be too bad, right?

Looking at this page gives me hope all I need to do is run rustup target add aarch64-unknown-linux-gnu and that should fix everything.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# rustup-init
warning: it looks like you have an existing installation of Rust at:
warning: /usr/bin
warning: rustup should not be installed alongside Rust. Please uninstall your existing Rust first.
warning: Otherwise you may have confusion unless you are careful with your PATH
warning: If you are sure that you want both rustup and your already installed Rust
warning: then please reply `y' or `yes' or set RUSTUP_INIT_SKIP_PATH_CHECK to yes
warning: or pass `-y' to ignore all ignorable checks.
error: cannot install while Rust is installed

Continue? (y/N)

Oh come on, there’s got to be a better way. And there is. There is some support for rust cross compilation in crossdev, although it requires some setting up. Nothing complicated, however: just add an ebuild to your crossdev overlay and install it.

Looks like this ebuild’s version is behind that of the rust ebuild. I’ll just bump it manually, hopefully it’s ok. And looks like it wants to download something off the net during installation, guess I’ll disable the sandbox as well.

Seems to have built just fine, but now the linking fails. Turns out I had to set RUST_CROSS_TARGETS and rebuild rust after all. Everything seems fine now.

Misfortune 8: Firefox

I don’t want to deal with clang right now, so I’ll go for a GCC build. Apparently the build process requires libclang anyway, to generate rust bindings for C++ code. Well, fine. The problem is, the current ebuild determines libclang’s location by calling llvm-config and for some reason this name resolves to llvm-config that’s installed in my ARM filesystem. Let’s throw a quick fix at the problem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

diff --git a/var/db/repos/gentoo/www-client/firefox/firefox-115.0.2.ebuild b/var/db/repos/gentoo/www-client/firefox/firefox-115.0.2.ebuild
index 6fabb6b..fe186cf 100644
--- a/var/db/repos/gentoo/www-client/firefox/firefox-115.0.2.ebuild
+++ b/var/db/repos/gentoo/www-client/firefox/firefox-115.0.2.ebuild
@@ -819,7 +819,7 @@ src_configure() {
                --without-ccache \
                --without-wasm-sandboxed-libraries \
                --with-intl-api \
-               --with-libclang-path="$(llvm-config --libdir)" \
+               --with-libclang-path="$(${CBUILD:-${CHOST}}-llvm-config --libdir)" \
                --with-system-nspr \
                --with-system-nss \
                --with-system-zlib \

Builds fine now.

Misfortune 9: preserved-rebuild

So the system finally builds, but there are lots of messages about @preserved-rebuild. Basically, it means there are some libraries that have been updated, but there are programs that may still rely on the old versions.

I ran the rebuild process, which turned out to include ~60 packages. But once it was done, I saw the same messages prompting me to rebuild everything again. Weird, maybe it’s some portage bug?

I found this topic on Gentoo forums, so at least I’m not the only one with this problem.

In the end I removed preserved_libs_registry, now portage doesn’t want to rebuild anything. I should probably use revdep-rebuild, but there doesn’t seem to be any integration with crossdev and I don’t want to deal with this right now.

Eh, it’ll be fine.

Conclusion

The system should work now, but whether it actually does is a separate question. : )

Time to throw everything on an SD card and find out.