Collaborating within a Codebase

Heroics in software development must be examined with a healthy dose of cynicism. While yes, sometimes short bursts of concentrated personal contributions become necessary to meet a deadline, these circumstances usually emerge from unexpected inconveniences or misunderstandings. If releases by heroics become the norm, we burden our software team with an unsustainable development process.

A common cause of added frantic work stems from treating the development of software and the release of software as two separate entities. Multiple development teams built outstanding improvements over the last months and each team runs a stable new version of the software. Unfortunately, when we consolidate all that effort for a public release, some changes turn out to be incompatible.

Our company dedicates subject matter experts from each team to work with our operations personnel to get the application working for the upcoming release date and employ all their metaphorical duct-tape, spit, and prayers. The amount of effort put into this time-sensitive build arouses the suspicion of our quality assurance team, who naturally find flaws in our deliverable.

With tensions running high, we decide what flaws we can live with, which ones we cannot. Our subject matter experts apply their craft once again to fix the designated bugs and hand over their result to QA. Our quality team gives the thumbs-up and we ship on time thanks to the heroic efforts of all involved. To avoid any regressions for future releases, we integrate our heroic bug fixes back into the development teams' code base. Unfortunately, our team members were working hard for the next release and some changes turn out be incompatible...

Working continuously

Rather than releasing a single high-effort impeccable version, we build software on two virtues: frequent iteration and continuous improvement. By refining a product over time, we may better respond to changing requirements, evolving technology, and approaching deadlines. To ship high-quality updates, we optimize our development process for frequent releases, and should we find flaws during the release procedure, our development teams provide improvements quickly and without introducing new (or old) errors.

A solid development and release process allows us to deliver our product with confidence. The more efficiently we can address unwanted behavior, the earlier we may ship our software to gather vital user insights. If we're forced to keep software behind closed doors until we're "done", we build an entire product lifecycle on (potentially misguided) assumptions. User feedback reveals what works, what doesn’t, and what was never needed in the first place. The sooner our software sees use, the sooner it becomes useful.

Besides gathering customer insights, teams that don't release software don't generate revenue. Every paying customer funds the further development of our product. The more paying customers we aggregate at an early stage, the faster we can grow the team, scale the product, and expand our market share. More importantly, the sooner our software makes money, the less pressure on our depleting bank accounts.

In order to make "ship fast, ship frequently" sound more business savvy, we refer to this strategy as CI/CD. Since any serious professional movement requires an acronym to be considered industry standard, this counter-intuitively conflated initialism that stands for three continuous practices:

• Continuous Integration
• Continuous Deployment
• Continuous Delivery

CI/CD insinuates automation. We build an environment that encourages developers to integrate code frequently, and have our system automatically build, test, and deploy code. CI/CD shifts the responsibility of mindless repetitive tasks to computers. Running tests, packaging build artifacts, signing binaries. The more processes we automate, the more time our engineers can spend on solving domain problems and writing business critical code.

Part I of Engineering Collaboration - Collaborating within a Codebase - focuses on the workflows for a complete CI/CD cycle. Each chapters covers a topic within that cycle to help teams to ship code fast, and ship frequently.

Continuous Integration

The first problem we encountered during the introductory example was our development teams working on different aspects of the software in their own isolated worlds. While every team produced (presumably) stable software in isolation, after consolidating all the new changes, we broke the build. The more people involved and the more work done, the further the different worlds diverge, and the more error-prone, tedious, and time-intensive task of creating a stable release candidate.

If an organization has a Continuous Integration (CI) process in place, all teams and team members integrate their work into the shared source code. The state of the source code reflects the state of the software to be released. Automated tests verify the integration of the source code and detect errors as quickly as possible. The fundamental goal of CI is to automatically catch problematic changes as early as possible by continuously assembling and testing complex and rapidly evolving ecosystems.

The more frequently engineers integrate changes, the greater the improvements of CI. CI offers verifiable and timely proof that our code changes can progress to the next stage. We don't need to hope that all contributors are careful, responsible, and thorough; we can guarantee the working state of our application at various points in time. By frequently checking small batches of revisions we reduce the overhead of integrating changes into the product.

We'll discuss continuous integration topics across the chapters Merge Requests, Code Reviews, and Testing.

Continuous Deployment

After integrating our code changes, we remotely build the application and run additional tests against the new binary. While CI verifies the changes to our source code, Continuous Deployment (CD) confirms the expected behavior of our compiled machine code.

Because continuous integration focuses purely on developer workflows, we tend to anchor the entirety of CI/CD to engineering. Thus, when talking about continuous deployment, the conversation shifts towards automated testing with self-service infrastructure in pseudo production environments. While important, deploying to a managed test environment is only the first step. Most software requires an automated and a manual thumbs-up before we ship it.

Engineers don't build tech in a silo. We talk to customers that reported an error. We prioritize product features with product owners. We design the user inputs and UI feedback flows with our UX colleagues. Before, during, and after writing code, we share our progress with all stakeholders and collectively build the software. A CI/CD mindset (and toolset) supports our aspects of collaborating iteratively on our products. The more we automate this process, the faster our feedback cycle.

Humans struggle to deliver consistent results across repetitive tasks. Building a software package manually requires us to follow a strict check list of compiling, copying, and renaming artifacts. Every so often, we miss a step and send our colleague a build that does not run due to misconfigured permissions or missing files. Computers, however, excel at performing the same steps over and over again. Automatically building, storing, and distributing software internally allows us to exchange updates with a high degree of velocity.

We discuss common strategies related to continuous deployment in the chapters Testing, Release Mechanisms, and Source and Artifact Management.

Continuous Delivery

Once we're happy with our changes, Continuous Delivery (CD) moves our tested package into production. Improving our software delivery effectiveness will improve our ability to work in small batches and incorporate customer feedback more rapidly.

As with the other CD (deployment), most conversations about delivery focus on smartly replacing code running in production. While CD (delivery) can cover that, that's considered the pinnacle of CI/CD. A pinnacle not appropriate for all software projects. It requires a high level of automation and testing, as well as a robust rollback process in case of issues. Hot loading to production is most suitable for internal deliveries or projects with a high degree of stability and a low risk of breaking changes.

Continuous delivery reduces the risk of the release process itself. Every public software update triggers a lot of moving parts: We run various release scripts to upload, sign, and publish artifacts and update our website with new download links and serve new documentation. Every step we manage to automate, reduces our overhead of releasing software. Less overhead allows us to release more often and increase our confidence in the process. Practice makes perfect.

Sometimes we discover errors that require an immediate fix. Whether we missed an edge case that deletes customer data, or learned of a critical security vulnerability in on of our libraries, we need to act fast and with competence. Robust continuous delivery allows us to be effective during emergencies. Opposite to people, high-running emotions and raised voices do not affect software automation. It also removes hard dependencies on key personnel during time-sensitive events.

We write customer facing release notes and creating marketing material. automatically, taking screenshots automatically. Immediate distribution and delivery with low overhead processes. Docs, Marketing, Sales, these all have to happen, but they cannot be blockers. Our CI/CD process needs to enable and provide information for these. Release notes, Screenshots, videos, tutorials, etc.

The terms Continuous Deployment and Continuous Delivery have been used interchangeably to such a degree that there's a probable chance different teams use these terms vice versa. We are not partial to any order. In this book, we deploy to test (stage) and deliver to production (client).

The chapters Source and Artifact Management, Documentation, and Release Strategies discuss various topics of continuous delivery. We discuss how we verify our changes and detect errors quickly in the chapter Monitoring and Observability.

Shift Left

Quote

"Shifting left is about paying attention to the low fuel warning light on an automobile versus running out of gas on the highway."
- Unknown, Misquoted

The term originates from a visual representation of a typical development cycle. We map the development steps in chronological order from left to right - left being the conception of a product, right being the release of the final product. The mindset of "shifting left" encourages us to detect and resolve problems in the earliest (leftest) possible stage.

The cost needed to fix an error grows exponentially the further right we detect the error. With every step we involve more people and generate additional communication overhead. An engineer can often fix a regression discovered by a failing test by themselves in minutes while thy're working on the implementation. If discovered by a customer in production code, that customer reaches out to our support staff. The support engineer verifies the problem with the customer and relays their issues upstream. The people in charge of our products documentation update the "Known Issues" category for the latest build.

The development team investigates the cause for the regression - the magnitude of the task depending on release frequency and source code management - and assigns time to fix it. Equipped with customer logs and our support's case report and engineer tries to debug the issue, but might have to reach out to the customer for clarification. After integrating a fix, we verify the new (old) behavior and restart our release process to ship a patch.

The above example describes a scenario with two measurable cost comparisons. Every software leader may calculate the offset betwen investing into a CI/CD shift left environment and the cost of not doing so.