Announcing a Shiny New Johnny Deps

Here's Johnny

We’ve released a shining^Wshiny new version of Johnny Deps, our dependency manager for Go. The new version is simpler, cleaner, and more powerful. It also has a shorter name, jd, instead of being called johnny_deps. It’s not backwards compatible, but it’s pretty similar.

Why Rewrite?

We rewrote because we found a few shortcomings in the older version. These can be summarized as two main items:

  1. Wrapping go get. The go get tool is great for its intended purpose, but it has no notion of package versioning. As a result, it’s sometimes at odds with versioned packages. A simple example of this is if you want to pin a package to an older version and there’s a backwards-incompatible change. If you run go get, it will try to pull the code from its repositories, but it’ll fail because there will be a compilation error. There are other problems as well – go get can be a bit heavyhanded sometimes, and isn’t really something you want to run again and again after you’ve fetched your source code and set it up. It’s really better suited for fetching source for the first time in a non-versioned way.
  2. Not enough functionality. We ended up with build.sh scripts in all of our repositories. These glued together some tasks such as fetching the source, setting all the dependencies to the correct versions, setting our --build-version variable, and the other types of things we discussed in our previous blog post on creating 100% reproducible builds. A lot of this was boilerplate that we wanted to move out of that script and combine with the dependency and version management.

The New Johnny Deps

A summary of the new Johnny Deps would need to mention the following points:

  1. It is named jd so it’s clear that it’s a different tool. It’s also nicer to type.
  2. It is designed to support a common set of workflows in a single convenient tool. The default action is to get and build, but you can also ask it to only get, or to update the Godeps file. In most cases for our own development workflow, we’ll be using jd instead of a mixture of go and other commands.
  3. It doesn’t use go get anymore. It just uses git. It doesn’t make much sense to wrap a wrapper around git, so we do it directly. This gives us a lot more control and eliminates some overhead and unwanted behaviors. It’s true that it only works with Git. That’s because this tool is for our use first and foremost. We have a policy that all code must be in our own Git repositories. We do not build binaries in the ways that Go’s tools support (e.g. fetching unknown source code from potentially random and untrusted repositories scattered around the Internet). Those workflows are nice for open-source collaboration but do not suit our business needs. The jd tool is designed for our requirements. We’ve open-sourced it because we believe lots of other people are likely to have similar needs.
  4. It works recursively. Whereas previously a Godeps file was expected to include every package and its SHA at a top-level, now each repository’s Godeps file only lists its immediate dependencies and their versions. Dependencies are expected to have their own Godeps files. If it’s not in a Godeps file, the source won’t be fetched, and the program won’t compile. Dependencies are verified; if there’s a diamond-shaped dependency graph and conflicting/different dependency versions are listed in the respective Godeps files, the tool will print an error and exit non-zero.
  5. It is flexible so you can include custom build instructions in an executable if you want.

Please take a look at the documentation. We hope you find the new Johnny Deps helpful and welcome your contributions!