Module gcp_node_upgrade 

Triggers rollouts of Materialize instances when GKE upgrades the node pools they are running on.

GKE automatically upgrades node pools (e.g. to roll out new node images), and this cannot be disabled. With the blue-green upgrade strategy, GKE first creates replacement (green) nodes, then cordons all of the existing (blue) nodes, then drains them in batches (respecting pod disruption budgets), and finally deletes them after a soak period of up to seven days. Left alone, this would evict or force-delete environmentd and clusterd pods, causing an outage.

This module instead moves the pods with the standard graceful rollout machinery before GKE gets around to deleting the nodes:

• A Pub/Sub subscriber listens for GKE cluster notifications (UpgradeEvents) and arms a node pool when it starts upgrading. Since notifications can be missed (e.g. while orchestratord is restarting), the GKE API is additionally polled at startup and periodically thereafter, arming any watched pool with an upgrade in progress.

• While a pool is armed, its blue-green upgrade phase is polled from the GKE API. Once the phase reports that all blue nodes have been cordoned (DRAINING_BLUE_POOL or later), each Materialize instance with environmentd or clusterd pods on the cordoned nodes gets a forced rollout, triggered by setting the materialize.cloud/force-rollout annotation on the v1 Materialize resource. The new generation of pods can only be scheduled onto the green nodes (the blue nodes are unschedulable), and the old generation is torn down gracefully once the new one is ready.

Arming on upgrade notifications rather than triggering on any cordon avoids spurious (and expensive) rollouts when a node is cordoned for reasons that don’t mean the node is going away, e.g. an administrator debugging a node. Waiting for the cordoning phase to complete before triggering ensures the new generation cannot be scheduled onto a blue node that simply hadn’t been cordoned yet and will still be drained.

Structs

Config

Functions

run

Runs the GCP node upgrade watcher forever. Errors are logged and retried.