Hi, my name is Ashley and Iām part of the core engineering team.
As we continue developing Vega, weāre obviously thinking ahead to when people around the world will want to run Vega nodes. Not all of them will be running the same hardware and operating systems as we do, and we need to cater for that.
Rather than having multiple machines, each on different hardware and running a different operating system, and each compiling Vega for their particular hardware and operating system combination, itās far preferable to choose one combination and compile multiple times in such a way that each resulting program can be run on a particular operating system and hardware combination. This process is called cross-compiling.
Enter Golang cross-compiling
The Vega core node software is written in Golang, and the helpful Golang developers have done their best to make cross-compiling easy.
To see the impressive variety of supported operating system and hardware combinations, run go tool dist list
. The list should include many operating systems (Android, Darwin (MacOSX), Linux, several BSD flavours, and Windows) and several hardware architectures (386, AMD64, ARM, ARM64, and some MIPS flavours).
It looks like weāre done. Just run GOOS=myOS GOARCH=myCPU go build ./app
and go home. And that would be the case, if we didnāt need cgo
.
Oh, cgo
cgo
is used to inject C or C++ code into Golang code. Hereās a simple C example:
package main
// #include <stdio.h>
//
// void sayHi() {
// printf("Hello, embedded C!\n");
// }
import "C"
func main() {
C.sayHi()
}
With Golang code that includes C/C++ code, a C/C++ compiler is needed in addition to the Golang one. Unfortunately, C/C++ compilers are not nearly as friendly with cross-compiling as the Golang one is. A hardware-architecture-specific, operating-system-specific C/C++ compiler is needed for each hardware architecture and operating system combination. Our go build
command starts to grow:
env \
CGO_ENABLED=1 \
CC=my-OS-specific-C-compiler \
CXX=my-OS-specific-Cplusplus-compiler \
GOOS=myOS \
GOARCH=myCPU \
\
go build ./myapp
At this point, you might be wondering why we didnāt consider removing all the C/C++ code from Vega. If only it were so simple! Although Vega itself contains no embedded C or C++ code, it depends on go-ethereum
, which currently depends on duktape
, which needs cgo
. There is a go-ethereum
ticket to make duktape
an optional dependency, but in the meantime weāre going to need to Deal With Ittm.
xgo to the rescue
Rather than reinvent the wheel of tracking down and installing a set of C/C++ compilers, a quick online search revealed xgo
, which provides pre-created Docker container images packed full of C and C++ compilers for various hardware architectures and operating systems.
While our use of private Github dependencies did prevent xgo
from working out-of-the-box, a quick look at its main build script revealed the magic incantations necessary to be able to run a Docker container and compile for Linux and MacOSX. Our build command has grown into a docker run
command plus go build
command inside.
docker run --rm -ti \
-v "$HOME/.ssh:/root/.ssh" \
-v "$HOME/containergo/cache:/go/cache" \
-v "$HOME/containergo/pkg:/go/pkg" \
-v "$PWD:/go/src/project" \
-w /go/src/project \
-e CGO_ENABLED=1 \
-e CC=my-OS-specific-C-compiler \
-e CXX=my-OS-specific-Cplusplus-compiler \
-e GOOS=myOS \
-e GOARCH=myCPU \
-e GOCACHE=/go/cache \
--entrypoint go \
karalabe/xgo-latest:latest \
build ./myapp
Notably absent from the list of working operating systems was Windows. No amount of tweaking code, or dependency versions, or Go build options, was able to get the compilation to work. All I got was errors like āoledlg.h:428:3: error: unknown type name 'interface'
ā. Online searches also failed to help.
Somehow, though, I noticed that the xgo
Docker container image is based on Ubuntu 16.04. Since thatās pretty ancient, I thought Iād have a crack at updating it to Ubuntu 20.04. That process was simple but slow, even on an 8-CPU cloud-based VM I was using for the task, but eventually a shiny new Docker image was built and the moment of truth arrived.
After what seemed like half a lifetime of cross-compiling, the Windows build of Vega finished with no errors. The list of operating systems and hardware architectures Vega is now ready to have running nodes includes:
- Linux on 386, AMD64, ARM64
- Darwin (MacOSX) on AMD64
- Windows on 386, AMD64
A gift to the community
With so much time taken and effort made, Iād hate for other people to have to pointlessly go through the same process, so here are some resources to help avoid that:
-
GitHub - vegaprotocol/xgo: Go CGO cross compiler - Here you can find the updates and improvements made from the original
xgo
project. - Docker Hub - Here you can find Docker images build from the above Github repository. Images are available for Golang 1.11, 1.12, 1.13 and 1.14. For a summary of the updates and improvements, look at xgo-base.
Thatās all from me, I wish you luck in your Golang/cgo cross-compiling journey. If you have questions, please ask and Iāll be glad to try and help.