Release Operations
Our tests turn green and our code passes with flying colors. After planning, implementing, and refining our work, it's time to share our improvements with our users. However, while we're excited about shipping our changes, other team members are still actively working on their tasks, some of them slated for the next release, some of them in preparation for the one after. How do we prepare a release and continue development without stepping on each others toes?
When we have a stable version to release, we designate that snapshot as our "Release candidate". Once flagged, we build the product and unleash the full scope of our testing suites upon it: static analysis, small-, medium-, and large-scoped tests, benchmark tests, and in-house manual tests. If we discover any blocking issues, we address them and designate the fixed version as our updated release candidate.
Besides testing, candidates and the subsequent release require additional processing that development builds may skip. Signing and notarizing artifacts, rendering release notes, staging up-to-date documentation, or capturing videos and images of UI changes; the scope varies by software type, but ultimately we benefit from running these automatically per release once we've identified a candidate.
Green-lighting a candidate takes time and our developers push changes mercilessly. Depending on team size, the duration of verifying its behavior, and our release frequency, we adapt the mechanisms of referencing and processing a release candidate.
Releasing via Tags
Git allows us to reference specific commits via tags, static labels we can query to organize our automated workflows. To flag a release candidate we tag it via a defined pattern (e.g. rc1.2.3) to kick off our release candidate pipeline.
Unsuccessful release candidates require additional changes, and we follow our TBD practices of creating a development branch to work on a patch, create a pull request, merge our changes, and flag the resulting commit as a new release candidate rc1.23.5. If we're good to go, we tag the same commit with our release tag (e.g. r1.2.3 or v1.2.3) to set our release distribution pipeline into motion and publish our artifacts.
As our team grows and our integration frequency increases, tagging releases in the trunk becomes infeasible as our team merges unrelated changes. While we can orchestrate an integration delay for a team of one to three manually coding developers, this strategy degrades with an increase in man power or agent autonomy. We need a way to isolate the release process from ongoing development.
Releasing via Branches
Note the plural form in the title. Trunk-based development does not rely on a single long-lived branch that all releases merge into, so we should set aside the git-flow notion of the release branch. As covered in the introduction, trunk-based development defines optional release branches, but these are created for individual releases, instead of maintaining one permanent branch.
What does this look like in practice? Once we've identified a candidate snapshot, instead of tagging it, we spawn a branch off it and trigger our full pre-release automation and manual verification process. If we discover regressions or blockers, we fix them in main via a development branch.
If nobody integrated any other changes, we move the active release branch to the new version, otherwise we cherry pick the changes into it. We always resolve issues in main first and migrate it to the release to ensure that we resolved the problem in the product for future versions. Once the candidate process succeeded, we reference the commit in our release branch with a release tag to publish it.
Individual release branches isolate the release work from active development and facilitate teams with a high integration frequency or a lengthy release process. Our team members do not need to interfere with the delivery of our next product update as they integrate their work into the shared trunk. Microsoft shared their positive experience with this process and naturally dubbed it the Microsoft Release Flow) (surely not in response to other enterprise customers claiming the branching strategies Git Flow and GitHub Flow).
The primary immediate push-back I have heard to individual release branches comes in form of "don't I pollute my repository with all these release branches?". While repositories do accumulate these branches, the lightweight branch model of git removes any performance concerns and a one-line bash script may filter the output when listing the active branches.
Live at HEAD
When a team member integrates changes into the main branch, we run the full pre-release automation. Live at head treats every successful merge as a release candidate. This strategy represents the holy grail of TBD and CI/CD and requires robust and continuously improving automation for testing, analysis, documentation, and contract verification between teams. While release mechanism only makes sense for certain software products, the technical ability to deliver at any time is broadly worthwhile.
Besides pre-release validation, live at head systems require strong monitoring and disaster recovery strategies. Once a change has proven fragile in production we can rollback the latest release and redeploy an earlier build. We cover this topic in detail in the chapter Monitoring and Observability, but, in short, live at head deployments need automated rollback responses for failing systems with little (or no) human intervention.
Once any of our systems live reliably at HEAD, we have won continuous integration.
-
Good read? Unlock the rest of the chapter!
Engineering Collaboration is currently available as an Advanced Reading Copy for select readers.