I have been tasked with creating a new compiler for chipKIT. One of the big things this compiler must do is work on new versions of OS X. That means Big Sur. And that means supporting both x86_64 and arm64e (Apple silicon) architectures.
Now, compiling for x86_64 is not difficult - you just need an x86_64 Apple computer and crosstool-ng. And similarly if you want to compile for arm64e you just need an Apple-silicon based computer and, again, crosstool-ng.
Things get a lot more difficult, though, if you want to compile for both, and you only have an x86_64 computer (or in my case virtual machine), and even more complex if you want to result in a "fat" binary - that is a program that contains executable code for both architectures in one.
So, step one is to get crosstool-ng itself working. This is not as straight forward as just installing an app - you have to compile it from source, and to do that you need some extra tools that OS X doesn't have, and you need to fool the system into using them.
To start off you need the xtools command line tools. Those you install with:
xtool-select --install
and follow the prompts. That's simple enough.
Next you need Homebrew. Full instructions are on brew.sh but in short, from a terminal, run:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
And sit back and wait.
Once done you can install the required tools and libraries that you'll need:
brew install binutils gnu-sed gawk bash autoconf automake xz help2man libtool ncurses
And now grab the latest version of crosstool-ng (there is a version in Homebrew, but I like to always use the latest version as it then gives you access to the newest versions of GCC etc):
git clone https://github.com/crosstool-ng/crosstool-ng
Now the magic part - the bit that took me some time to figure out - how to get crosstool-ng to compile. First "bootstrap" it:
cd crosstool-ng ./bootstrap
It may take a while, but eventually it will get there and you'll be ready to configure it. Here's the bits you don't find online:
export OBJDUMP=/usr/local/Cellar/binutils/2.35.1_1/bin/gobjdump export OBJCOPY=/usr/local/Cellar/binutils/2.35.1_1/bin/gobjcopy export READELF=/usr/local/Cellar/binutils/2.35.1_1/bin/greadelf export MENU_LIBS="-L/usr/local/Cellar/ncurses/6.2/lib -lmenu" ./configure
Those first four lines tell the configure script where to find the particular extra tools you installed without polluting the rest of your environment. You could prepend those lines infront of the ./configure (omitting the export) if you like, but it makes for a very messy command. Better to split it into multiple lines and do it one at a time. By the way, you may need to change the version numbers in the paths above as time goes on.
Now you should have a configured system and be ready to compile:
make sudo make install
And you're done.
That was the easy bit. Now comes the real black magic - cross-compiling for arm64e.
OS X uses clang as the internal compiler. Unlike GCC, clang supports multiple target architectures from one single program. With GCC you normally have lots of different installations of the compiler for each target, such as x86_64-linux-gnu-gcc and aarch64-linux-gnu-gcc etc. With OS X you don't - you just have clang, and you provide a command line flag to specify what architecture to compile to. That's good, but not from the standpoint of crosstool-ng which expects the GCC way of doing things. So we need to fool it. And the way I chose is to make a set of wrapper scripts that look like the GCC compiler but actually just call the clang and llvm binaries instead.
So in ~/bin I created a set of script files, each with just one line in them:
arm64e-apple-darwin20-ar
#!/bin/bash ar "$@"
arm64e-apple-darwin20-as
#!/bin/bash as -arch arm64e "$@"
arm64e-apple-darwin20-g++
#!/bin/bash clang++ --target=arm64e-apple-darwin20 "$@"
arm64e-apple-darwin20-gcc
#!/bin/bash clang --target=arm64e-apple-darwin20 "$@"
arm64e-apple-darwin20-ld
#!/bin/bash ld -arch arm64e "$@"
arm64e-apple-darwin20-nm
#!/bin/bash nm -arch arm64e "$@"
arm64e-apple-darwin20-objcopy
#!/bin/bash /usr/local/Cellar/binutils/2.35.1_1/bin/objcopy "$@"
arm64e-apple-darwin20-objdump
#!/bin/bash objdump --mcpu=arm64e "$@"
arm64e-apple-darwin20-ranlib
#!/bin/bash ranlib "$@"
arm64e-apple-darwin20-strip
#!/bin/bash strip "$@"
And of course make them executable:
chmod 755 ~/bin/arm64e-apple-darwin20-*
Now you should have a working cross-compiler in a GCC style. You just need to ensure that ~/bin is in your path:
export PATH=~/bin/${PATH}
There's one more little caveat though: crosstool-ng has no clue what arm64e is. But that's no problem - a small tweak of a configuration file is all that's needed.
Find the file /usr/local/share/crosstool-ng/scripts/config.sub and look for the line arm64-*) and adjust it so it reads arm64-* | arm64e-*). It's owned by root, so you'll have to use sudo and your favourite terminal-based editor (nano, vi, etc) to edit it:
sudo vi /usr/local/share/crosstool-ng/scripts/config.sub
Now you should be able to do a "Canadian" cross-compilation for arm64-apple-darwin20 using crosstool-ng and it should all work.
Building fat binaries
This is where things get really fun. I know of no way to build fat binaries straight off. Instead what I have found you need to do is make two completely separate compiler trees - one for each architecture. That is, build a cross-compiler for x86_64-apple-darwin20 and another for arm64e-apple-darwin20 side-by-side. Then you need to take each program within that tree and merge them together into fat binaries (with lipo) building up a new, combined, output tree as you go.
There is probably some magic Apple tool for doing this, but it's not too hard to write a script to do it.
If you build the two compilers with the same settings you should end up with two trees where the only difference is the executable files. So we can nominate one tree as the "master" and the other as the "secondary" - we scan one tree (I'll use the native x86_64 one) and test what the file is. If it's an x86_64 executable then look for the same file in the secondary tree, then merge the two using lipo into a file in the output tree. If it's not an executable then just copy the file (unless it's a directory where we just make that directory). It would be good to preserve the file permissions too.
So let's have a go. Here's what I've come up with:
#!/bin/bash
# Master tree
SRC=HOST-x86_64-apple-darwin20
# Secondary tree
ALT=HOST-arm64e-apple-darwin20
# Output
OUT=HOST-fat-apple-darwin20
mkdir -p ${OUT}
# Scan the primary tree for directories and make them in the output
(cd $SRC && find . -type d) | while read FILE; do
mkdir -p "${OUT}/${FILE}"
PERM=$(stat -f "%p" "${SRC}/${FILE}")
chmod ${PERM} "${OUT}/${FILE}"
done
# Scan the primary tree for files and make them in the output
(cd $SRC && find . -type f) | while read FILE; do
TYPE=$(lipo -info "${SRC}/${FILE}" 2>/dev/null)
if [[ "${TYPE}" == *"is architecture: x86_64"* ]]; then # It's a binary
echo lipo -create "${SRC}/${FILE}" "${ALT}/${FILE}" -output "${OUT}/${FILE}"
lipo -create "${SRC}/${FILE}" "${ALT}/${FILE}" -output "${OUT}/${FILE}"
else
cp "${SRC}/${FILE}" "${OUT}/${FILE}"
fi
PERM=$(stat -f %p "${SRC}/${FILE}")
chmod ${PERM} "${OUT}/${FILE}"
done
# Scan the primary tree for links and make them in the output
(cd $SRC && find . -type l) | while read FILE; do
DEST=$(readlink "${SRC}/${FILE}")
DIR=$(dirname "${FILE}")
BNAME=$(basename "${FILE}")
(
cd "${OUT}/${DIR}"
ln -s "${DEST}" "${BNAME}"
)
done
There's probably a much better way of doing it, but it appears to have worked. All the executables say they're fat with both architectures in them, and they still work on the x86_64 system I can test them on.