Module hedge 

A Blob decorator that hedges slow get requests.

Established connections to the blob store occasionally die in ways that surface only after multiple seconds (a TCP reset after a hang, or a black hole), well before any client timeout fires. A get riding such a connection stalls everything downstream of it, while other connections on the same process serve the same store normally. The mitigation, endorsed by the major object stores for idempotent reads, is a hedged request: if the first get has not completed within a short delay, race a second one on a connection the first cannot have poisoned, and take whichever succeeds first.

Only get is hedged. All other Blob methods are forwarded to the primary handle untouched: writes, deletes, and restores have side effects, and lists are not latency-critical enough to justify racing a streaming interface. Extending hedging to any of them is forbidden.

The hedge handle must not share a connection pool (or DNS state) with the primary, otherwise the hedge can be handed a connection dying in the same event that stalled the primary, exactly when a hedge matters most. See crate::cfg::open_hedge_sibling for how that isolation is constructed per backend.

Hedging operates within a single retry_external attempt, before any failure surfaces. The retrying in retry_external, which is what recovers this failure class when hedging is off (at the cost of the full hang), stays untouched as the backstop. The governing principle for every race below: the primary’s outcome is authoritative, and the hedge is opportunistic, invisible unless it wins. Nothing here assumes callers retry: every branch of the race degrades to the outcome of the un-hedged get, delayed by at most one hedge delay, so a caller that treats a get error as fatal sees the same error it would have seen without hedging, at most that one delay later.

NOTE: enabling hedging largely suppresses the old fingerprints of the dead-connection class (client timeout counters, the SDK’s connection-poisoning log lines), because the hung request is cancelled before they trigger. The hedges_won counter is the replacement signal.

Structs

HedgeBudget 🔒

Bounds hedge amplification with two independent guards: the concurrency cap bounds memory held by raced gets, the token bucket bounds long-run request-rate/egress amplification (e.g. a store-wide brownout making every get slow, or large gets that legitimately exceed the delay, must not settle into hedging every request).

HedgeGuard 🔒

HedgedBlob

A Blob decorator that hedges slow get requests, per the module docs.

Enums

HedgeRefused 🔒

Why a hedge was not fired for a get that exceeded the delay.

HedgeSibling

The sibling handle a HedgedBlob runs hedge requests on, produced by crate::cfg::open_hedge_sibling.

Constants

BLOB_HEDGED_GET_BUDGET_RATIO 🔒

BLOB_HEDGED_GET_DELAY 🔒

BLOB_HEDGED_GET_ENABLED 🔒

BLOB_HEDGED_GET_MAX_CONCURRENT 🔒

BLOB_HEDGED_GET_WARM_INTERVAL 🔒

BUDGET_BURST_MICRO_TOKENS 🔒

Token-bucket capacity: 32 hedges.

HEDGE_COST_MICRO_TOKENS 🔒

The cost of one hedge in bucket tokens. Micro-token granularity keeps small budget_ratio values (down to 1e-6) from rounding to “never refill”.

SIBLING_OPEN_RETRY_INITIAL_BACKOFF 🔒

Backoff between sibling open attempts. The ceiling bounds a long outage to one attempt per minute per process (a credential resolution, a health-check get and a warning) and still re-arms within a minute of recovery.

SIBLING_OPEN_RETRY_MAX_BACKOFF 🔒

Functions

arm_then_warm 🔒

Installs the sibling into slot once opener yields one, then keeps it warm. See HedgedBlob::new_arming for the retry contract.

install 🔒

recheck_interval 🔒

How often an idle sibling loop re-reads the dyncfgs, so a flip takes effect without a restart: the warm interval, or its default while warming is set to 0.

warm 🔒

Keeps the sibling’s connection pool warm with periodic concurrent liveness gets, while hedging is enabled. A cold hedge can stall up to the connect timeout, which during correlated connection events is exactly when it must not. While hedging is disabled the warmer idles and the sibling sees no traffic at all, so a freshly enabled flag can find a cold pool for up to one warm interval plus a handshake. Hedges in that window are no better than no hedge, and never worse except for the grace window’s delay on a late primary error.

Type Aliases

HedgeSiblingOpener

Opens the sibling handle for a store. HedgedBlob::new_arming calls it until it returns something other than HedgeSibling::Unavailable, and relies on it to log the cause of each failed attempt. See crate::cfg::open_hedge_sibling.