Everyone wants faster Continuous Integration (CI). The quicker you can get from commit to test results, the better – but there’s a bit of tension here. A CI should start close to “first principles”, so that you know an entire build is good and repeatable. But the more work the CI is doing, the slower it will go.
I’ve mentioned before that I use a combination of stowage and container-based CI systems, and in professional life we’re very heavily invested in GitLab now (which is an excellent system in general). Containers are a good starting point for CI, because they’re very easy to set up, do some work, and then throw away again. Getting clean builds is very straightforward.
Today I released version 0.5.0 of stowage, which brings a few additional small features which really simplifies usage in CI.
Just to recap for those who haven’t used stowage before, it’s basically a really simple tool which can take container images and make them available as if they were local binaries. Here’s how I can take the official MS Azure container and turn it into a local command:
$ sudo stowage install microsoft/azure-cli --command=azure --entrypoint=azure azure installed $ azure --version 0.10.14 (node: 6.10.0)
The point here is not to replace package management in the system at all, but the end effect in terms of the user experience is very similar: what’s happening underneath is that stowage is creating a little wrapper around
docker run. It’s purely a convenience thing.
Why might this be useful for CI?
So, I find stowage useful in a couple of ways. Primarily, it means I can create “tools” (actually docker images) which I can distribute very easily. I can use these tools as a developer, or within the CI – but ideally I use them in both. Then I know that I have exactly the same tool chain across my team.
This is great for new developers; setting up a new environment is very simple. If I have docker and stowage installed, that’s all I need to start pulling down everything else.
A while ago, I had an insight that CI systems look an awful lot like new developers. A new developer doesn’t have an environment set up or the code available; in a similar sense, neither does the CI. You’re starting from a clean base.
I believe there’s a lot of correlation between the quality of onboarding experience for new developers, and the quality of a team’s CI process. When I’ve found teams with great onboarding processes, their CI process is also good: equally, when they struggle to integrate new developers, I’ve found the CI to be highly complex and fragile (if present at all). The quality of one really does predict the quality of the other.
So, it made sense to me that a tool I had created for the purpose of improving the developer experience would also be applicable to CI.
Some patterns, then!
So, first, having the main build tools be available as containers that you use as tools with stowage is a good plan. Ensuring that developers and CI uses the same tools is valuable. For example, rather than developers installing node and npm locally, why not use the official NodeJS container image?
$ sudo stowage install node:6 --command npm --entrypoint npm npm installed $ npm --version 3.10.10
In many CI systems based on containers, the use of a “build container” is commonplace. This is something I’ve recommended in the past when talking about Habitus, and it’s still a good move. However, it can lead to “kitchen sink” type images which have lots of different tools built in. This can make updating the image slower.
With stowage, you can think a little more in terms of composition of images. Within your main build image, you can run stowage install at the start of every build – since the images are already in the docker daemon’s image cache, this is a very quick process. You can then call out to the commands hosted in those images from your main build.
Potentially, you can also extend this to include dependencies for your build. For example, you could create an npm image that is based on the regular node image but adds in a package cache containing all the regular versions of the dependencies you need.
This is a bit of a poor man’s package cache pattern, but means that as well as distributing the known-good tools for your build chain, you also have the known-good dependencies. All available within containers, which can live within your system and allow you to function completely offline.
Does this work with hosted CI?
Sadly, for the most part, I think the answer is “No” right now – although I would love to hear from anyone who manages to do this. Primarily, stowage relies on relatively privileged access to the docker daemon, which is pretty incompatible with hosted CI.