Code Shipping - Continuous (development|Deployment)

1 - About

Continuous development reflects the habit of deploying new code as a series of small changes as soon as they are ready.

Continuous Deployment follows your tests (continuous integration )to push your changes to either a staging or production system.

3 - (Development|Deployment) Cycle

There is still a distinction between code in development and code that is ready to be deployed. Developers use the git version control system locally for their daily work, until the code is ready to push. The stable version for deployment is maintained using subversion (for historical reasons). When ready to be pushed, new code must first be merged with the stable version in the centralized repository, after which engineers can commit their changes into subversion.

Some of the factors going into the determination of the optimal deployment cycle are:

  • the cost of each deployment,
  • the probability and cost of errors,
  • the probability and value of incremental benefits,
  • the skill of the engineers involved,
  • and the culture of the organization.

Adding to the complexity of the decision is that many of these factors can be optimized, so the optimal cycle can change.

4 - Feature with large changes to the codebase

Ostensibly, rapid deployment is at odds with feature development that requires large changes to the codebase. The solution is to break down such changes into a sequence of smaller and safer ones, hidden behind an abstraction.

For example, consider the delicate issue of migrating data from an existing store to a new one. This can be broken down as follows:

  1. Encapsulate access to the data in an appropriate data type.
  2. Modify the implementation to store data in both the old and the new stores.
  3. Bulk migrate existing data from the old store to the new store. This is done in the background in parallel to writing new data to both stores.
  4. Modify the implementation to read from both stores and compare the obtained data.
  5. When convinced that the new store is operating as intended, switch to using the new store exclusively (the old store may be maintained for some time to safeguard against unforeseen problems).

Facebook has used this process to transparently migrate database tables containing hundreds of billions of rows to new storage formats. Development and Deployment at Facebook Dror Feitelson, Eitan Frachtenberg, Kent Beck

5 - Attribute

5.1 - Immediatly availabe to a a small subset of users

Deploying new code does not necessarily imply that it is immediately available to users. Facebook uses a tool called “Gatekeeper” to control which users see which features of the code. Thus it is possible for engineers to incrementally deploy and test partial implementations of a new service without exposing them to end users.

5.2 - A/B testing: Live experimentation

An important attribute of continuous deployment is that it facilitates live experimentation using A/B testing. The innovations that engineers implement are deployed immediately for real users to experience.

A/B testing is an experimental approach to finding what users want, rather than trying to elicit requirements in advance and writing specifications.

This enables a careful comparison of the new features with the base case (that is, the current site) in terms of their effect on user behavior [5].

While this typically involves only a small subset of users, at Facebook’s volume of activity even a very small subset quickly generates enough data to assess the impact of the tested features. Thus engineers can immediately identify what works in practice and what does not.

5.3 - Reduces the risk

Frequent deployments imply that each deployment introduces only a limited amount of new code. This reduces (but doesn’t eliminate) the risk that something will go wrong.

All commits are individually tested for regressions.

5.4 - Easily identify the source and solutions of problems

Another benefit of small and rapid deployments is that we can easily identify the source of and solutions to emerging problems: they’re most likely the most recently deployed changes in the code, and still fresh in engineers’ minds.

5.5 - No long-lived branches to merge

All front-end engineers at Facebook work on a single stable branch of the code, which also promotes rapid development, since no effort is spent on merging long-lived branches into the trunk.

6 - Tools

7 - Documentation / Reference

code/shipping/continuous_deployment.txt · Last modified: 2017/09/07 12:35 by gerardnico