Kubernetes allowed us to clearly define responsibilities between the product and ops teams. It provided the primitives necessary - such as Services, Deployments and ConfigMaps - to roll our software out in a fast, predictable manner. The product teams could manage their own deployments while the ops teams offered the monitoring services needed to ensure that their deployments went smoothly.

All code at Thought Machine lives in a single monorepo, where we maintain code in multiple languages (mainly Python, Go and Java), as well as our Kubernetes resource files and infrastructure code. This allows the infrastructure team to make sweeping changes across the board in an atomic way. In the past, we maintained a separate repository for Puppet modules, but coordinating changes and deployments between multiple repos was difficult and error prone.

We have a strict code review policy where any change must be reviewed by at least one person (including a senior developer depending on the sensitivity of the code), a policy we’ve enforced from the beginning. It’s proven to be an amazing way to educate others, discuss designs or to simply notify other engineers of what’s going on. As soon as a PR is raised your changes will be picked up by a build agent and all of the affected tests will be run in the background (we use specialised on-premise servers for this).

Once your branch has been merged into master, we determine the Docker images that have been affected, re-build them and tag them with a unique hash based on the contents of the image. We then move on to template our Kubernetes object definitions (explained below).

Since we treat Docker labels as immutable, updating our Deployment YAML files every time an image changed would be painful. Instead, we template these values in at the end of the build. An example of a committed deployment might look like: