I’ve been hacking in golang since before version 1.4, and the speed at which my builds finished has been mostly trending downwards. Let’s look into the reasons and some fixes. TL;DR click-bait title: “Get 4x faster golang builds with this one trick!”.
Here are the three reasons my builds got slower:
The compiler
Before version 1.5, the compiler was written in C but with that release, it moved to being pure golang. This unfortunately reduced build performance quite measurably, even though it was the right decision for the project.
There have been slight improvements with newer versions, however Google’s focus has been improving runtime performance instead of build performance. This is understandable since they want to save lots of electricity dollars at scale, which is not as helpful for smaller shops where the developer iteration cycle is the metric to optimize.
This could still be improved if folks wanted to put in the effort. A gcc style -O0 option could help. The sad thing about this whole story is that “instant” builds were a major marketing feature of early golang presentations.
Project size
Over time, my main project (mgmt) has gotten much bigger! It compiles in a number of libraries, including etcd, prometheus, and more! This naturally increases build time and is a mostly unavoidable consequence of building cool things and standing on giants!
This mostly can’t be helped, but it can be mitigated…
Dependency caching
When you build a project and all of its dependencies, the unchanged dependencies shouldn’t need to be rebuilt! Unfortunately golang does a great job of silently rebuilding things unnecessarily. Here’s why…
When you run a build, golang will attempt to re-use any common artefacts from previous builds. If the golang versions or library versions don’t match, these won’t be used, and the compiler will redo this work. Unfortunately, by default, those results won’t be saved, causing you to waste CPU cycles every time you test!
When the intermediate results are kept, they are found in your $GOPATH/pkg/. To save them, you need to either run go install (which makes a mess in your $GOPATH/bin/, or you can run go build -i. (Thanks to Dave for the tip!)
The sad part of this story is that these aren’t cached by default, and stale results aren’t discarded by default! If you’re experiencing slow builds, you should rm -rf $GOPATH/pkg/ and then go build -i. After one successful build, future builds should be much faster!
Example
james@computer:~/code/mgmt$ time go build # before
real 0m28.152s
user 1m17.097s
sys 0m5.235s
james@computer:~/code/mgmt$ time go build -i # after
real 0m8.129s
user 0m12.014s
sys 0m0.839s
Debugging
If you want to debug what’s going on, you can always run go build -x.
Blame!
I don’t like assigning blame, but this feels like a case of the golang tools being obtuse, and the man pages being non-existent. The golang project has a lot of maturing to do to integrate sanely with a stock GNU environment:
man go build could exist and provide useful informationgo build --helpgo help build could provide more useful information--help)$XDG_CACHE_HOME.Hope this helped improve your golang experience! I always knew something was going in in $GOPATH/pkg/, but I think it’s pretty absurd that I only fully understood it now. My builds are about 4x faster now. :)
Happy Hacking,
James
You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.
Your comment has been submitted and will be published if it gets approved.
Click here to see the patch you generated.
Comments
Nothing yet.
Post a comment