Module caught_up 

Support for checking whether clusters/collections are caught up during a 0dt deployment.

During a zero-downtime upgrade the new environmentd boots read-only and reports “ready to promote” once its clusters have caught up with the leader generation. A task spawned by Coordinator::spawn_caught_up_check_task runs that check on an interval (see WITH_0DT_DEPLOYMENT_CAUGHT_UP_CHECK_INTERVAL). We call one such run a “tick”, and the term is used throughout this module.

Each tick the task reads the leader’s Persist collections, then asks the coordinator to capture local state and issue hydration requests. The task awaits the replies, classifies the clusters and applies the stability gate. None of those waits run on the coordinator loop.

A point-in-time caught-up check is not enough on its own: a crash- or OOM-looping replica can momentarily look hydrated and caught-up, and cutting over right then drops us straight into a crashing replica. On top of the per-tick caught-up classification we therefore run a stability gate. A caught-up cluster is ready only once every replica is Online and every non-exempt replica has had its non-ignored collections hydrated for a configurable period, capped for young replicas by observe_stability.

Each replica reports its hydration times in its mz_compute_hydration_times_per_worker introspection log. The log lives in the replica process. It survives environmentd restarts, because a restarted environmentd reconciles with the same replica processes and they keep compatible dataflows, so a DDL-triggered restart does not restart the period. A dataflow that changed rehydrates and restarts it. The log dies with the process, so a restarted replica must hydrate again and establish a new observation period, subject to the young-replica cap.

Hydration times come from the replica’s clock, while the period is measured against environmentd’s. A replica clock that runs ahead only lengthens the wait, but one that runs behind shortens the period by the skew. We rely on hosts keeping their clocks in sync.

A replica without introspection logging cannot report hydration times. It only has to be Online. Leader-relative exemptions are defined by LeaderHydration::replica_has_unhydrated_collection.

Structs

CaughtUpCheck 🔒

Inputs owned by one background check.

CaughtUpCheckContext

Context needed to check whether clusters/collections are caught up.

CaughtUpCheckRequest

Requests local controller state without waiting for external reads.

CaughtUpCluster 🔒

A caught-up cluster, as the stability gate needs to know it.

CaughtUpSnapshot 🔒

The cluster classification and stability inputs for one tick.

ClusterSnapshot 🔒

LeaderHydration 🔒

The leader’s hydration reports, read from its mz_compute_hydration_times.

LeaderSnapshot 🔒

StabilityGate 🔒

The part of the caught-up check that runs in its own task: it asks replicas for their hydration times and applies the stability gate.

StabilityObservation 🔒

Outcome of folding a cluster’s replica health.

Enums

ClusterCaughtUpStatus 🔒

How a cluster relates to the 0dt caught-up check on a given tick.

CollectionType 🔒

ReplicaHealth 🔒

What the stability gate knows about one replica on a given tick.

ReplicaTarget 🔒

What the coordinator knows about a replica of a caught-up cluster, and so whether the stability gate has to ask it.

StabilityBlocker 🔒

Why a caught-up cluster is being held back by the stability gate on a given tick. Recorded so we can log the cause.

Constants

STABILITY_QUERY_TIMEOUT 🔒

How long the stability gate waits for an internal query.

Functions

hydration_query 🔒

Returns a query for when a replica last finished hydrating an export.

observe_stability 🔒

Folds the health of a cluster’s replicas into the gate’s verdict.

replica_health 🔒

Interprets the rows of a hydration_query.