DevBlog - Cross-Compiling Vega with GO, CGO and XGO

Hi, my name is Ashley and Iā€™m part of the core engineering team.

As we continue developing Vega, weā€™re obviously thinking ahead to when people around the world will want to run Vega nodes. Not all of them will be running the same hardware and operating systems as we do, and we need to cater for that.

Rather than having multiple machines, each on different hardware and running a different operating system, and each compiling Vega for their particular hardware and operating system combination, itā€™s far preferable to choose one combination and compile multiple times in such a way that each resulting program can be run on a particular operating system and hardware combination. This process is called cross-compiling.

Enter Golang cross-compiling

The Vega core node software is written in Golang, and the helpful Golang developers have done their best to make cross-compiling easy.

To see the impressive variety of supported operating system and hardware combinations, run go tool dist list. The list should include many operating systems (Android, Darwin (MacOSX), Linux, several BSD flavours, and Windows) and several hardware architectures (386, AMD64, ARM, ARM64, and some MIPS flavours).

It looks like weā€™re done. Just run GOOS=myOS GOARCH=myCPU go build ./app and go home. And that would be the case, if we didnā€™t need cgo.

Oh, cgo

cgo is used to inject C or C++ code into Golang code. Hereā€™s a simple C example:

package main

// #include <stdio.h>
//
// void sayHi() {
//   printf("Hello, embedded C!\n");
// }
import "C"

func main() {
	C.sayHi()
}

With Golang code that includes C/C++ code, a C/C++ compiler is needed in addition to the Golang one. Unfortunately, C/C++ compilers are not nearly as friendly with cross-compiling as the Golang one is. A hardware-architecture-specific, operating-system-specific C/C++ compiler is needed for each hardware architecture and operating system combination. Our go build command starts to grow:

env \
	CGO_ENABLED=1 \
	CC=my-OS-specific-C-compiler \
	CXX=my-OS-specific-Cplusplus-compiler \
	GOOS=myOS \
	GOARCH=myCPU \
	\
	go build ./myapp

At this point, you might be wondering why we didnā€™t consider removing all the C/C++ code from Vega. If only it were so simple! Although Vega itself contains no embedded C or C++ code, it depends on go-ethereum, which currently depends on duktape, which needs cgo. There is a go-ethereum ticket to make duktape an optional dependency, but in the meantime weā€™re going to need to Deal With Ittm.

xgo to the rescue

Rather than reinvent the wheel of tracking down and installing a set of C/C++ compilers, a quick online search revealed xgo, which provides pre-created Docker container images packed full of C and C++ compilers for various hardware architectures and operating systems.

While our use of private Github dependencies did prevent xgo from working out-of-the-box, a quick look at its main build script revealed the magic incantations necessary to be able to run a Docker container and compile for Linux and MacOSX. Our build command has grown into a docker run command plus go build command inside.

docker run --rm -ti \
	-v "$HOME/.ssh:/root/.ssh" \
	-v "$HOME/containergo/cache:/go/cache" \
	-v "$HOME/containergo/pkg:/go/pkg" \
	-v "$PWD:/go/src/project" \
	-w /go/src/project \
	-e CGO_ENABLED=1 \
	-e CC=my-OS-specific-C-compiler \
	-e CXX=my-OS-specific-Cplusplus-compiler \
	-e GOOS=myOS \
	-e GOARCH=myCPU \
	-e GOCACHE=/go/cache \
	--entrypoint go \
	karalabe/xgo-latest:latest \
		build ./myapp

Notably absent from the list of working operating systems was Windows. No amount of tweaking code, or dependency versions, or Go build options, was able to get the compilation to work. All I got was errors like ā€œoledlg.h:428:3: error: unknown type name 'interface'ā€. Online searches also failed to help.

Somehow, though, I noticed that the xgo Docker container image is based on Ubuntu 16.04. Since thatā€™s pretty ancient, I thought Iā€™d have a crack at updating it to Ubuntu 20.04. That process was simple but slow, even on an 8-CPU cloud-based VM I was using for the task, but eventually a shiny new Docker image was built and the moment of truth arrived.

After what seemed like half a lifetime of cross-compiling, the Windows build of Vega finished with no errors. The list of operating systems and hardware architectures Vega is now ready to have running nodes includes:

  • Linux on 386, AMD64, ARM64
  • Darwin (MacOSX) on AMD64
  • Windows on 386, AMD64

A gift to the community

With so much time taken and effort made, Iā€™d hate for other people to have to pointlessly go through the same process, so here are some resources to help avoid that:

  • GitHub - vegaprotocol/xgo: Go CGO cross compiler - Here you can find the updates and improvements made from the original xgo project.
  • Docker Hub - Here you can find Docker images build from the above Github repository. Images are available for Golang 1.11, 1.12, 1.13 and 1.14. For a summary of the updates and improvements, look at xgo-base.

Thatā€™s all from me, I wish you luck in your Golang/cgo cross-compiling journey. If you have questions, please ask and Iā€™ll be glad to try and help.

6 Likes

Iā€™m looking forward to getting the trial Raspberry Pi-net up and running

2 Likes

Can I ask whatā€™s the purpose of:
-v "$HOME/.ssh:/root/.ssh" \ ?

It is at least suspicious to mount a directory that contains private keys into an unknown image.

Hi @lelettrone, thanks for the question, and welcome to the Vega community.

:male_detective: Youā€™re completely correct that itā€™s suspicious to be mounting a directory containing private keys in an unknown image.

The reason this is necessary for Vega is our private github source repositories. When we run go build or go install, dependencies are fetched from github, and will fail unless we take some extra steps.

  1. For our repos, we switch from https to ssh:

    git config --global url."git@github.com:vegaprotocol".insteadOf \
        "https://github.com/vegaprotocol"
    
  2. We add -v "$HOME/.ssh:/root/.ssh" to inject the credentials into the build container.

In case youā€™re about to say that itā€™s not best practice to inject credentials in this way, I agree, but this is only a build container, and the final image is built by copying the final compiled binary using COPY --from=builder in order to ensure it contains no credentials.

If youā€™re building an open-source app, or at least all your dependencies are publicly accessible, then you shouldnā€™t need -v "$HOME/.ssh:/root/.ssh" when using xgo.

Hi @Ashley,
for sure if you trust the owner of the image there is no problem at all.

Maybe in a real case I would clone the repo in a clean container where I feel confident to share my credentials and make an extra ā€œCOPYā€ into the build containerā€¦ but, ok, I understand this is a blog post and has to be as simple as possible.
Indeed I thank you for the useful reading and for the clarification. :wink:

1 Like