Struct Migration

struct Migration {
    source_version: Version,
    target_version: Version,
    deploy_generation: u64,
    system_objects: BTreeMap<SystemObjectDescription, ObjectInfo>,
    migration_shard: ShardId,
    config: BuiltinItemMigrationConfig,
}

Context of a builtin schema migration.

Fields

source_version: Version

The version we are migrating from.

Same as the build version of the most recent leader process that successfully performed migrations.

target_version: Version

The version we are migration to.

Same as the build version of this process.

deploy_generation: u64

The deploy generation of this process.

system_objects: BTreeMap<SystemObjectDescription, ObjectInfo>

Information about all objects in the system.

migration_shard: ShardId

The ID of the migration shard.

config: BuiltinItemMigrationConfig

Additional configuration.

Implementations

impl Migration

async fn run(self, steps: &[MigrationStep]) -> Result<MigrationRunResult>

fn validate_migration_steps(&self, steps: &[MigrationStep])

Sanity check the given migration steps.

If any of these checks fail, that’s a bug in Materialize, and we panic immediately.

fn plan_migration(&self, steps: &[MigrationStep]) -> Plan

Select for each object to migrate the appropriate migration mechanism.

fn plan_forced_migration(&self, mechanism: Mechanism) -> Plan

Plan a forced migration of all objects using the given mechanism.

async fn upgrade_migration_shard_version(&self)

Upgrade the migration shard to the target version.

async fn migrate_evolve( &self, objects: &[SystemObjectDescription], ) -> Result<()>

Migrate the given objects using the Evolution mechanism.

async fn migrate_evolve_one( &self, object: &SystemObjectDescription, ) -> Result<()>

async fn migrate_replace( &self, objects: &[SystemObjectDescription], ) -> Result<BTreeMap<GlobalId, ShardId>>

Migrate the given objects using the Replacement mechanism.

async fn try_get_or_insert_replacement_shards( &self, ids_to_replace: &BTreeSet<GlobalId>, persist_write: &mut WriteHandle<Key, ShardId, Timestamp, StorageDiff>, persist_read: &mut ReadHandle<Key, ShardId, Timestamp, StorageDiff>, ) -> Result<Option<BTreeMap<GlobalId, ShardId>>>

Try to get or insert replacement shards for the given IDs into the migration shard, at target_version and deploy_generation.

This method looks for existing entries in the migration shards and returns those if they are present. Otherwise it generates new shard IDs and tries to insert them.

The result of this call is None if no existing entries were found and inserting new ones failed because of a concurrent write to the migration shard. In this case, the caller is expected to retry.

async fn open_migration_shard( &self, diagnostics: Diagnostics, ) -> (WriteHandle<Key, ShardId, Timestamp, StorageDiff>, ReadHandle<Key, ShardId, Timestamp, StorageDiff>)

Open writer and reader for the migration shard.

async fn open_migration_shard_since( &self, diagnostics: Diagnostics, ) -> SinceHandle<Key, ShardId, Timestamp, StorageDiff, i64>

Open a SinceHandle for the migration shard.

fn update_fingerprints( &self, migrated_items: &BTreeSet<SystemObjectDescription>, ) -> Result<BTreeMap<SystemObjectDescription, String>>

Update the fingerprints for migrated_items.

Returns the new fingerprints. Also asserts that the current fingerprints of all other system items match their builtin definitions.

async fn cleanup(&self) -> Result<(BTreeSet<ShardId>, BoxFuture<'static, ()>)>

Perform cleanup of migration state, i.e. the migration shard.

Returns a list of shards to finalize, and a Future that must be run after the shard finalization has been durably enqueued. The Future is used to remove entries from the migration shard only after we know the respective shards will be finalized. Removing entries immediately would risk leaking the shards.

We only perform cleanup in leader mode, to keep the durable state changes made by read-only processes a minimal as possible. Given that Materialize doesn’t support version downgrades, it is safe to assume that any state for versions below the target_version is not needed anymore and can be cleaned up.

Note that it is fine for cleanup to sometimes fail or be skipped. The size of the migration shard should always be pretty small, so keeping migration state around for longer isn’t a concern. As a result, we can keep the logic simple here and skip doing cleanup in response to transient failures, instead of retrying.