Expand description
A Blob decorator that hedges slow get requests.
Established connections to the blob store occasionally die in ways that
surface only after multiple seconds (a TCP reset after a hang, or a black
hole), well before any client timeout fires. A get riding such a
connection stalls everything downstream of it, while other connections on
the same process serve the same store normally. The mitigation, endorsed by
the major object stores for idempotent reads, is a hedged request: if the
first get has not completed within a short delay, race a second one on a
connection the first cannot have poisoned, and take whichever succeeds
first.
Only get is hedged. All other Blob methods are forwarded to the
primary handle untouched: writes, deletes, and restores have side
effects, and lists are not latency-critical enough to justify racing a
streaming interface. Extending hedging to any of them is forbidden.
The hedge handle must not share a connection pool (or DNS state) with the primary, otherwise the hedge can be handed a connection dying in the same event that stalled the primary, exactly when a hedge matters most. See crate::cfg::open_hedge_sibling for how that isolation is constructed per backend.
Hedging operates within a single retry_external attempt, before any
failure surfaces. The retrying in retry_external, which is what
recovers this failure class when hedging is off (at the cost of the full
hang), stays untouched as the backstop. The governing principle for
every race below: the primary’s outcome is authoritative, and the hedge
is opportunistic, invisible unless it wins. Nothing here assumes callers
retry: every branch of the race degrades to the outcome of the un-hedged
get, delayed by at most one hedge delay, so a caller that treats a get
error as fatal sees the same error it would have seen without hedging,
at most that one delay later.
NOTE: enabling hedging largely suppresses the old fingerprints of the
dead-connection class (client timeout counters, the SDK’s
connection-poisoning log lines), because the hung request is cancelled
before they trigger. The hedges_won counter is the replacement signal.
Structs§
- Hedged
Blob - A Blob decorator that hedges slow
getrequests, per the module docs.
Enums§
- Hedge
Sibling - The sibling handle a HedgedBlob runs hedge requests on, produced by crate::cfg::open_hedge_sibling.