Data - Migration

> (Data|State) Management and Processing

1 - About

Migrating data from an existing store to a new one.

See also: Database - Migration (Deployment|Versioning|Change Management)

Advertising

3 - Example

The migration of data can be broken down as follows:

  1. Encapsulate access to the data in an appropriate data type.
  2. Modify the implementation to store data in both the old and the new stores.
  3. Bulk migrate existing data from the old store to the new store. This is done in the background in parallel to writing new data to both stores.
  4. Modify the implementation to read from both stores and compare the obtained data.
  5. When convinced that the new store is operating as intended, switch to using the new store exclusively (the old store may be maintained for some time to safeguard against unforeseen problems).
Facebook has used this process to transparently migrate database tables containing hundreds of billions of rows to new storage formats.