100% Reproducible Builds in Go

by Baron Schwartz on 10 Sep 2013

One of the nice things about Go is statically compiled binaries, which don't have external dependencies. There are lots of reasons this is a great thing. On the other hand, as we've mentioned before, Go packages aren't versioned.

This results in a minor irritation that isn't fixed natively in the Go-provided toolchain: how can I constrain which versions of external packages are imported into my application? And, given that this isn't one of the things that is supplied with Go itself, how can I a) determine what source code and which versions of packages were used to build an app, and b) reproduce that exactly for troubleshooting and debugging?

terra-cotta

At VividCortex, we've developed a couple of tools and a technique to solve this problem. We've open-sourced most of the tools already, and in this blog post I'll explain how we tie these together into a fully functioning solution.

First of all, we use a technique that we call Godeps-in-Git. This is shorthand slang for one of the two approaches mentioned in the Johnny Deps documentation. This means that we commit a full, explicit Godeps file into our applications' Git repos. These contain the Git SHA of every non-core (i.e. not part of the Go standard library) package.

To generate this list, we use the generate_deps tool provided in Johnny Deps.

Then we use a build script instead of calling “go build” directly. In addition to executing johnny_deps, it has a pre-build step that fetches the application repo's Git SHA, writes out a temp file, and then performs the build. The script looks like this:

echo "package main" > godeps.go
echo "func init() {" >> godeps.go
echo " BuildSHA = \"# Built $(date) with $(go version) at $(git rev-parse HEAD)\"" >> godeps.go
echo "}" >> godeps.go

What does this do? It writes out an init() function which sets a global variable called BuildSHA. Now, we add a commandline flag to our application, called –build-sha. If this flag is present, we'll print out the value of BuildSHA:

Config.BoolVar(&WantSHA, "build-sha", WantSHA, "Print the packages and versions in this build")
// ...
if WantSHA {
    fmt.Println(BuildSHA)
    os.Exit(0)
}

The result is that I can execute the app with the –build-sha flag and get output like the following:

 $ ./app --build-sha
    # Built Mon Sep 9 20:53:06 EDT 2013 with go version go1.1 darwin/amd64 at 1f9398aebb652f7d494c783f83393b8cab7c54bc

If I want to reproduce that build, I can:

$ git checkout 1f9398aebb652f7d494c783f83393b8cab7c54bc
$ johnny_deps

And I'm done. Now I have exactly the source code, in the app's repository as well as every external package, that was used to build the app. This is pretty similar to “pip freeze,” if you're familiar with that.

There's also a little helper in our build script, which we invoke manually. It runs generate_deps to find the app's dependencies, pulls them from Git, and commits them to Godeps again. We use this to “upgrade” an app's Godeps file, so we don't stay pinned to old versions of dependencies.

Hopefully you find these tools and practices helpful in creating reproducible builds for your Go applications. Let us know if you have any questions or suggestions!

pic credit

comments powered by Disqus