Skip to main content

Module hedge

Module hedge 

Source
Expand description

A Blob decorator that hedges slow get requests.

Established connections to the blob store occasionally die in ways that surface only after multiple seconds (a TCP reset after a hang, or a black hole), well before any client timeout fires. A get riding such a connection stalls everything downstream of it, while other connections on the same process serve the same store normally. The mitigation, endorsed by the major object stores for idempotent reads, is a hedged request: if the first get has not completed within a short delay, race a second one on a connection the first cannot have poisoned, and take whichever succeeds first.

Only get is hedged. All other Blob methods are forwarded to the primary handle untouched: writes, deletes, and restores have side effects, and lists are not latency-critical enough to justify racing a streaming interface. Extending hedging to any of them is forbidden.

The hedge handle must not share a connection pool (or DNS state) with the primary, otherwise the hedge can be handed a connection dying in the same event that stalled the primary, exactly when a hedge matters most. See crate::cfg::open_hedge_sibling for how that isolation is constructed per backend.

Hedging operates within a single retry_external attempt, before any failure surfaces. The retrying in retry_external, which is what recovers this failure class when hedging is off (at the cost of the full hang), stays untouched as the backstop. The governing principle for every race below: the primary’s outcome is authoritative, and the hedge is opportunistic, invisible unless it wins. Nothing here assumes callers retry: every branch of the race degrades to the outcome of the un-hedged get, delayed by at most one hedge delay, so a caller that treats a get error as fatal sees the same error it would have seen without hedging, at most that one delay later.

NOTE: enabling hedging largely suppresses the old fingerprints of the dead-connection class (client timeout counters, the SDK’s connection-poisoning log lines), because the hung request is cancelled before they trigger. The hedges_won counter is the replacement signal.

Structs§

HedgedBlob
A Blob decorator that hedges slow get requests, per the module docs.

Enums§

HedgeSibling
The sibling handle a HedgedBlob runs hedge requests on, produced by crate::cfg::open_hedge_sibling.