Module mz_catalog::durable::upgrade

source ·
Expand description

This module contains all the helpers and code paths for upgrading/migrating the Catalog.

We facilitate migrations by keeping snapshots of the objects we previously stored, and relying entirely on these snapshots. These snapshots exist in the mz_catalog_protos crate in the form of catalog-protos/protos/objects_vXX.proto. By maintaining and relying on snapshots we don’t have to worry about changes elsewhere in the codebase effecting our migrations because our application and serialization logic is decoupled, and the objects of the Catalog for a given version are “frozen in time”.

Note: The protobuf snapshot files themselves live in a separate crate because it takes a relatively significant amount of time to codegen and build them. By placing them in a separate crate we don’t have to pay this compile time cost when building the catalog, allowing for faster iteration.

You cannot make any changes to the following message or anything that they depend on:

  • Config
  • Setting
  • FenceToken
  • AuditLog

When you want to make a change to the Catalog you need to follow these steps:

  1. Check the current CATALOG_VERSION, make sure an objects_v<CATALOG_VERSION>.proto file exists. If one doesn’t, copy and paste the current objects.proto file, renaming it to objects_v<CATALOG_VERSION>.proto.
  2. Bump CATALOG_VERSION by one.
  3. Make your changes to objects.proto.
  4. Copy and paste objects.proto, naming the copy objects_v<CATALOG_VERSION>.proto. Update the package name of the .proto to package objects_v<CATALOG_VERSION>;
  5. We should now have a copy of the protobuf objects as they currently exist, and a copy of how we want them to exist. For example, if the version of the Catalog before we made our changes was 15, we should now have objects_v15.proto and objects_v16.proto.
  6. Rebuild Materialize which will error because the hashes stored in src/catalog-protos/protos/hashes.json have now changed. Update these to match the new hashes for objects.proto and objects_v<CATALOG_VERSION>.proto.
  7. Add v<CATALOG_VERSION> to the call to the objects! macro in this file, and to the proto_objects! macro in the mz_catalog_protos crate.
  8. Add a new file to catalog/src/durable/upgrade which is where we’ll put the new migration path.
  9. Write upgrade functions using the two versions of the protos we now have, e.g. objects_v15.proto and objects_v16.proto. In this migration code you should not import any defaults or constants from elsewhere in the codebase, because then a future change could then impact a previous migration.
  10. Add an import for your new module to this file: mod v<CATALOG_VERSION-1>_to_v<CATALOG_VERSION>;
  11. Call your upgrade function in run_upgrade().
  12. Generate a test file for the new version:
    ⓘ
    cargo test --package mz-catalog --lib durable::upgrade::tests::generate_missing_encodings -- --ignored

When in doubt, reach out to the Surfaces team, and we’ll be more than happy to help :)

Modules§

Macros§

Enums§

  • MigrationAction 🔒
    Describes a single action to take during a migration from V1 to V2.

Constants§

Functions§