Working on changes

Building software as a team becomes deceptively complex very fast. All individual code changes need to be compatible with everybody else's changes and the more people that work on shared code, we higher the chance we create conflicting contributions. In order to retain our sanity, professional software engineers collaborate within a codebase using a version control system. By comparing the source changes over time, we can write code without vandalizing each other’s work.

Every aspect of our organization benefits from tracking changes over time, be it source code, documentation, and other digital assets. Collaborating within a Codebase assumes foundational understanding of version control principles and we will not be rehashing how any particular version control tool works. The practices covered in this part of the book can be achieved with any available open-source and commercial tooling.

While this book provides tools to scale teams and source code indefinitely, we'll find that teams outgrow their current processes every time the number of contributors doubles in size. Whether we grow from a single developer to two, from the thirties to the sixties, or from the hundreds to twice that, we need to adjust certain tools and processes.

Trunk-Based Development

Most version control systems do not enforce a particular workflow, in fact, they advertise the ability for teams to create their own processes and branching models. However, over the years our industry discovered that most software teams encounter similar problems and certain workflows have proven themselves as "industry standard". One such standard is Trunk-Based Development.

The key differentiator of Trunk-based development is to enable Continuous Integration, Deployment, and Delivery. While we can set up a Trunk-Based Development workflow for any version control tool, due to the large adoption of Git across the industry, I'll explain the concepts using Git terms, semantics, and visual representations.

TBD differentiates three types of branches: The titular trunk, typically called main, a single branch from which all other branches sprout from and merge to. Secondly, short-lived development branches for engineers to do their work. We create a new development branch for every task we work on and delete it once we merged our code to main. Lastly, we have optional release branches. We'll skip these for now and discuss them in detail in the chapter Release Mechanisms.

We keep our trunk, main, stable. While the code in main may include work in progress and not be ready for public consumption, we can always compile, build, and package the software from any commit in our trunk. It may seem counter-intuitive at first, but we achieve this by integrating code frequently into our main branch. Only when merging our changes on a regular basis can we establish a solid foundation for our CI/CD process. By promoting small, incremental changes, we minimize conflicts and reduce the risk of introducing errors into the codebase.

All software work is done in development branches. The first step of writing any code changes is to create a new development branch from the latest commit in main. During development, we commit often to the development branch and push our changes to back up our work. No code lives solely on our hard drive, as we're ever only one power surge away from frying our machine and crying over lost effort.

When we completed our work, we merge the changes back into main and delete the development branch. The larger the team, the higher the integration frequency should be. We resist any urges to create long-lived development branches. When mapped to the (sub)tasks of our assigned work, our development branch ideally lives for a couple of hours up to a couple of days.

Responsibility

The world of software engineering created branching models other than TBD. Most rely on one or multiple nested long-living dev branches to integrate changes incrementally across the org chart. These non-committal intermediate branches work as a buffer to protect the integrity of the main branch and not accidentally break the build for other teams.

While well intentioned, our engineers still merge incompatible changes and just don't know about it. They pass the buck downstream. These branching models rely on department heads (or dedicated branch masters) to integrate changes into parent branches. This practice leads to unnecessary handoffs and dependencies between teams. A key part of TBD and shifting left is for engineers to take responsibility of their work and ensure changes behave as intended in the larger scope of the product.

Automation tools build and test the software before and after our developers merge their changes into main. When the same person writes the source code, the tests, the documentation, and integrates the changes themselves, chances are pretty high that we catch unexpected behavior during the merging process.

While it's an engineers individual responsibility to extend, run, and pass the automation suite, it's the team's responsibility to define what tools to run at which point of the integration journey. Some tasks are mandatory - as subject matter experts provide tools to ensure organizational requirements and security policies - but as an organization we put in the effort to reduce their impact. Annoying mandatory automation leads to teams routinely bypassing them, effectively rendering the processes mute.

Members of autonomous teams accompany changes through inception, implementation, and release. Teams who release changes autonomously identify more strongly with the product they worked on and the organization they worked for. The saying you build it, you own it increases the connection to the feature and supports organic transparency, feedback loops, and learning. Engineers deliver value to the customer, not submit implementation suggestions for our department.

Even though our engineers take responsibility for integrating their work into the shared trunk, out teams define the processes and tools to guide the integration. Mistakes happen, even our most experienced personnel make them. We cannot rely on people doing "the right thing" or, inversely, not doing the "the wrong thing". One common method to regulate integrations is to declare our trunk a protected branch.

Protected branches authorize only specified code owners to apply changes and prevent unapproved integrations. In TBD, we typically protect our trunk. Every change to main needs to be introduced via a development branch and merged using the UI of our version control platform. Nobody but code owners may apply and push changes directly to main. Limiting the authorized people fends off accidental or unintended commits that alter critical parts of a project.

Should we decide to protect a branch, we offer a visible option to override said protection. The tool is meant to guard against accidental breaking changes, not to introduce red tape. Creating hard dependencies on key personnel backfires as soon as our team needs to deploy an urgent hotfix while the code owner is out sick, on vacation, or at the pub.

Engineers that take responsibility of their integration with the help of mature CI/CD systems report higher employee satisfaction. According to the study done by Accelerate, continuous processes increase the trust in our testing and deployment pipeline, thus increasing the confidence in the work we deliver. This confidence reduces the uncertainty of hand-offs, relieving stress during work hours and thus leads to more effective downtime with family and friends.

Feature Flags

The source code of any published software lives in a perpetual dyad of polishing the current state for the upcoming release and introducing new features for the release after. In order to keep momentum, ship frequently, and not get in each other's way, we make use of two workflow mechanisms. We discuss one of these - release branches - in the chapter Release Mechanisms. The other mechanism is called Feature Flags, a technique that allows us to disable specific parts of our application.

Feature flags toggle our software's behavior depending on the context and who's working with it. Internally, this allows our teams to test and refine new features without interfering with the active expected behavior. Our recipients activate the testing feature locally by using a special command, akin to cheat-codes in video games, or setting values in dedicated configuration files. It's a proven way of sharing work in progress with relevant stakeholders and conceal it from the wider user base.

When distributing our software, we may rely on remotely toggled feature flags to activate new behavior for a subset of users. When we gradually roll out changes to our product and disable any feature behaving unexpectedly, we can test software features in a controlled and scalable manner without having to deploy new code to our application. We cover the topic of remotely toggling features in more detail in the chapter Release Strategies.

The scale of changes

Size matters. Trunk-based development thrives on frequent integrations of reasonable proportions. The longer we isolate ourselves in our respective development branches, the more difficult our merging process becomes. Other people's changes turn out to be incompatible with some of the work we did, or we introduced a regression early on, but haven't run our CI/CD tools for weeks.

When working with development branches, we consider two units of scale. Time passed since we spawned our branch and the volume of changes contained within our branch. Our working branch becomes more outdated with every change merged by others. The longer we remain detached from the trunk, the further our working application diverges from the source of truth.

Every line of code we change introduces potential conflicts with others. The more lines we touch, the higher the chance we create incompatible edits we need to review and resolve. the In the worst case scenarios, the time needed to review our massive development branch exceeds the team's average integration time of changes. Meaning, by the time we're ready to merge our changes, they've become outdated again due to new code changes in our trunk. The likelihood of this scenario increases with the volume of our changes.

Ideally, we spend between an hour to about four days working in our development branch, but realistically every team member will end up breaking this rule eventually. Most engineers can share war stories of them spending two weeks hunting down a bug that resulted in two lines of code changes. Rather than following the letter of law, the spirit of TBD dictates that no engineer spends more than three days actively writing source code without integrating the changes back to the trunk.

With every code change in a development branch, we increase the effort to keep that branch up-to-date with our trunk. Thus, with increasing volume, we need to integrate our changes faster and more frequently. This apparent paradox forces us to plan out an implementation strategy for large-scale changes, and, if necessary, split the change into sensible coherent packages to be integrated separately.

Focused engineering

Unfortunately, in the real world our problems seldomly emerge in a convenient fashion. From large feature requests, over major refactors, to architectural changes, large tasks require more than three days of work and thus exceed what we consider a reasonable development branch lifetime. To make use of TBD and CI/CD we chunk the extensive task into several self-contained subtasks.

For every subtask we write the source code, extend the test suite, and compose any needed documentation. We then integrate the completed partial work after passing the CI automation and guard any work-in-progress with feature flags. Introducing a large change via multiple smaller integrations reduces the effort involved and increases development velocity by catching errors early on in a reduced scope.

How developers split their tasks into separate integrations is predominantly based on experience, and the decision rests with the implementing engineer.

Even with small tasks we may find ourselves accidentally creating oversized development branches when we fall victim to another folly, distractions. While working on our feature we identify a bug which seems innocuous enough and we decide to fix it. We continue working on our task, when a day later a downstream dependency becomes brittle. Turns out the first bug masked a larger issue in the codebase and we go investigate...

Development branches isolate a set of changes, namely the task we're working on. When created, we resist any urge to add unrelated edits, such as refactors of code smells, renaming of variables or methods, or manual whitespace changes. These distractions not only add to our development time, but pollute our branch with unnecessary noise when comparing the differences to our trunk.

Should we come across a "distraction" we deem necessary to fix immediately, we create a new dedicated development branch and do so within its confines. Inherently consistent branches streamline the integration process and communicate the intent of the changes. If we inadvertently introduced unexpected behavior, a clean history with dedicated development branches supports future engineers in fixing the problem.

An engineers responsibility extends beyond the purely syntactical implementation. We ensure our changes work with other people's as expected and our changes work in the product as expected. We reduce any burden of integrating our changes and ensure future colleagues understand the intent behind them. We delivery iteratively and we deliver continuously.