DevBlog - Cross-Compiling Vega with GO, CGO and XGO

Hi, my name is Ashley and I’m part of the core engineering team.

As we continue developing Vega, we’re obviously thinking ahead to when people around the world will want to run Vega nodes. Not all of them will be running the same hardware and operating systems as we do, and we need to cater for that.

Rather than having multiple machines, each on different hardware and running a different operating system, and each compiling Vega for their particular hardware and operating system combination, it’s far preferable to choose one combination and compile multiple times in such a way that each resulting program can be run on a particular operating system and hardware combination. This process is called cross-compiling.

Enter Golang cross-compiling

The Vega core node software is written in Golang, and the helpful Golang developers have done their best to make cross-compiling easy.

To see the impressive variety of supported operating system and hardware combinations, run go tool dist list. The list should include many operating systems (Android, Darwin (MacOSX), Linux, several BSD flavours, and Windows) and several hardware architectures (386, AMD64, ARM, ARM64, and some MIPS flavours).

It looks like we’re done. Just run GOOS=myOS GOARCH=myCPU go build ./app and go home. And that would be the case, if we didn’t need cgo.

Oh, cgo

cgo is used to inject C or C++ code into Golang code. Here’s a simple C example:

package main

// #include <stdio.h>
//
// void sayHi() {
//   printf("Hello, embedded C!\n");
// }
import "C"

func main() {
	C.sayHi()
}

With Golang code that includes C/C++ code, a C/C++ compiler is needed in addition to the Golang one. Unfortunately, C/C++ compilers are not nearly as friendly with cross-compiling as the Golang one is. A hardware-architecture-specific, operating-system-specific C/C++ compiler is needed for each hardware architecture and operating system combination. Our go build command starts to grow:

env \
	CGO_ENABLED=1 \
	CC=my-OS-specific-C-compiler \
	CXX=my-OS-specific-Cplusplus-compiler \
	GOOS=myOS \
	GOARCH=myCPU \
	\
	go build ./myapp

At this point, you might be wondering why we didn’t consider removing all the C/C++ code from Vega. If only it were so simple! Although Vega itself contains no embedded C or C++ code, it depends on go-ethereum, which currently depends on duktape, which needs cgo. There is a go-ethereum ticket to make duktape an optional dependency, but in the meantime we’re going to need to Deal With Ittm.

xgo to the rescue

Rather than reinvent the wheel of tracking down and installing a set of C/C++ compilers, a quick online search revealed xgo, which provides pre-created Docker container images packed full of C and C++ compilers for various hardware architectures and operating systems.

While our use of private Github dependencies did prevent xgo from working out-of-the-box, a quick look at its main build script revealed the magic incantations necessary to be able to run a Docker container and compile for Linux and MacOSX. Our build command has grown into a docker run command plus go build command inside.

docker run --rm -ti \
	-v "$HOME/.ssh:/root/.ssh" \
	-v "$HOME/containergo/cache:/go/cache" \
	-v "$HOME/containergo/pkg:/go/pkg" \
	-v "$PWD:/go/src/project" \
	-w /go/src/project \
	-e CGO_ENABLED=1 \
	-e CC=my-OS-specific-C-compiler \
	-e CXX=my-OS-specific-Cplusplus-compiler \
	-e GOOS=myOS \
	-e GOARCH=myCPU \
	-e GOCACHE=/go/cache \
	--entrypoint go \
	karalabe/xgo-latest:latest \
		build ./myapp

Notably absent from the list of working operating systems was Windows. No amount of tweaking code, or dependency versions, or Go build options, was able to get the compilation to work. All I got was errors like “oledlg.h:428:3: error: unknown type name 'interface'”. Online searches also failed to help.

Somehow, though, I noticed that the xgo Docker container image is based on Ubuntu 16.04. Since that’s pretty ancient, I thought I’d have a crack at updating it to Ubuntu 20.04. That process was simple but slow, even on an 8-CPU cloud-based VM I was using for the task, but eventually a shiny new Docker image was built and the moment of truth arrived.

After what seemed like half a lifetime of cross-compiling, the Windows build of Vega finished with no errors. The list of operating systems and hardware architectures Vega is now ready to have running nodes includes:

  • Linux on 386, AMD64, ARM64
  • Darwin (MacOSX) on AMD64
  • Windows on 386, AMD64

A gift to the community

With so much time taken and effort made, I’d hate for other people to have to pointlessly go through the same process, so here are some resources to help avoid that:

That’s all from me, I wish you luck in your Golang/cgo cross-compiling journey. If you have questions, please ask and I’ll be glad to try and help.

6 Likes

I’m looking forward to getting the trial Raspberry Pi-net up and running