Module gcp_node_upgrade 

Triggers rollouts of Materialize instances when GKE upgrades the node pools they are running on.

GKE automatically upgrades node pools (e.g. to roll out new node images), and this cannot be disabled. With the blue-green upgrade strategy, GKE first creates replacement (green) nodes, then cordons all of the existing (blue) nodes, then drains them in batches (respecting pod disruption budgets), and finally deletes them after a soak period of up to seven days. Left alone, this would evict or force-delete environmentd and clusterd pods, causing an outage.

This module instead moves the pods with the standard graceful rollout machinery before GKE gets around to deleting the nodes:

• A Pub/Sub subscriber listens for GKE cluster notifications (UpgradeEvents) and arms a node pool when it starts upgrading. Since notifications can be missed (e.g. while orchestratord is restarting), the GKE API is additionally polled at startup and periodically thereafter, arming any watched pool with an upgrade in progress.

• While a pool is armed, its blue-green upgrade phase is polled from the GKE API. Once the phase reports that all blue nodes have been cordoned (DRAINING_BLUE_POOL or later), each Materialize instance with environmentd or clusterd pods on the cordoned nodes gets a forced rollout, triggered by setting the materialize.cloud/force-rollout annotation on the v1 Materialize resource. The new generation of pods can only be scheduled onto the green nodes (the blue nodes are unschedulable), and the old generation is torn down gracefully once the new one is ready.

Arming on upgrade notifications rather than triggering on any cordon avoids spurious (and expensive) rollouts when a node is cordoned for reasons that don’t mean the node is going away, e.g. an administrator debugging a node. Waiting for the cordoning phase to complete before triggering ensures the new generation cannot be scheduled onto a blue node that simply hadn’t been cordoned yet and will still be drained.

Structs

ArmedPool 🔒

ArmedPools 🔒

Config

GcpApiClient 🔒

A minimal client for the GCP REST APIs used here (Pub/Sub and GKE), authenticating via Application Default Credentials (in particular, GKE workload identity).

Enums

BlueGreenPhase 🔒

The blue-green upgrade phase of a node pool, from the GKE API.

PoolCheckOutcome 🔒

Constants

DATA_PLANE_POD_SELECTOR 🔒

Label selector matching the pods which must be moved before their node goes away: environmentd and clusterd pods, which are moved between generations by the rollout machinery. Balancerd and console pods are stateless deployments and can be drained normally.

GKE_NODE_POOL_LABEL 🔒

The node label GKE uses to record which node pool a node belongs to.

MAX_ARMED_DURATION 🔒

Safety valve: disarm pools which have been armed for longer than this. GKE caps the total soak time of a blue-green upgrade at seven days, so a pool armed for longer than this is not going to see any more drains from the upgrade that armed it.

Functions

check_armed_pool 🔒

Checks the upgrade progress of a single armed node pool, triggering rollouts for affected instances once the blue pool is fully cordoned.

maybe_trigger_rollout 🔒

Triggers a forced rollout of the given Materialize instance, unless one is already in progress or was triggered very recently.

poll_gke_for_upgrades 🔒

Returns the watched node pools which the GKE API reports as having an upgrade in progress.

run

Runs the GCP node upgrade watcher forever. Errors are logged and retried.

scan_loop 🔒

Polls the armed node pools, and the GKE API for pools that should be armed, triggering rollouts for the instances on pools whose blue nodes have all been cordoned. Never returns.

subscriber_loop 🔒

Pulls GKE cluster notifications from the Pub/Sub subscription forever, arming node pools when they start upgrading.

upgrading_node_pool 🔒

If the given Pub/Sub message attributes describe an UpgradeEvent for a watched node pool of our cluster, returns the node pool name.