Struct PoolInner 

struct PoolInner {

    budget_bytes: AtomicU64,
    rss_target_bytes: AtomicU64,
    regions: Vec<Region>,
    extent_arena: Arc<ExtentArena>,
    store: ExtentStore,
    full_hint: AtomicBool,
    parked_extents: Mutex<ParkedExtents>,
    parked_classes: AtomicU64,
    file_release: Mutex<FileRelease>,
    queues: [Mutex<VecDeque<Weak<ChunkMeta>>>; 4],
    extent_queue: Mutex<VecDeque<ExtentEntry>>,
    live_chunks: AtomicU64,
    extent_residents: AtomicU64,
    enforcing: Mutex<()>,
    enforce_pending: AtomicBool,
    counters: Counters,
    spill: Spill,
}

Available on Unix and crate feature pool only.

The shared state behind every Pool handle. One per process in practice; Pool clones and chunk handles share it through an Arc, so it lives until the last handle and spill thread release it.

Lock order: a chunk’s state mutex may be held while taking any of the leaf locks (the eviction queue, the extent_queue, the spill queue, the region slot allocators, and the file store’s per-class slot allocators, which alloc, free, and hole punching take) but never the reverse. The enforcement and backing scans additionally drop the queue guard before trying a chunk’s state lock, so no path holds a queue lock while waiting on chunk state. Scans only ever try_lock a chunk they pop, with one exception: a demotion re-locks its victim with a blocking lock to commit after the unlocked write. That wait holds no other chunk or queue lock. From budget enforcement it holds the enforcing single-flight mutex, which is only ever try_locked, so no thread waits on it. Hence enforce_compressed_cap must never run while its caller holds any chunk’s state lock: two such callers demoting each other’s chunks would deadlock on the re-lock. The admitting read’s victim steal is the one place a chunk’s state lock is held while probing another chunk’s, and the victim is only ever try_locked, so two admitters stealing toward each other skip instead of deadlocking.

Reads copy out under the chunk’s state lock, the same lock eviction takes, so there is no reader-side count and no reader the evictor must account for. The exception is a plain read of an extent demoted to file, which copies out after the unlock, sound per Home::File in extent.rs.

Fields

budget_bytes: AtomicU64

Resident-bytes target, enforced against evictable bytes (resident minus heap-backed, which no eviction can reclaim). Atomic so a running pool can be retuned in place (operator-driven budget changes) without orphaning live handles, which share this value through their Arc<PoolInner>.

rss_target_bytes: AtomicU64

Ceiling on the pool’s total RSS: slots (the budget) plus warm free slots plus compressed-resident extents. The compressed tier’s capacity derives as max(0, rss_target - budget - warm cap); zero (the default) collapses the tier, paging every extent out as soon as it is written.

regions: Vec<Region>

One region per entry of SIZE_CLASSES, same order.

extent_arena: Arc<ExtentArena>

The arena backing extents. Shared with every live Extent, whose drop returns its slot.

store: ExtentStore

Where RSS-target enforcement sends arena extents: the swap device, or a file store that demoted extents keep alive.

full_hint: AtomicBool

File mode: set when a demotion pass found the store unable to take any extent it probed, cleared by a committed demotion or a freed file slot, or by enqueueing an extent whose class the store can place. While set, inline callers run no demotion pass, and each spill thread runs one at most every SPILL_PARK_TIMEOUT.

NOTE: setting and clearing race. A pass can set the hint after a concurrent free cleared it, and demotion then stays deferred until the next free, commit, or enqueue of a placeable extent, or the next spill thread retry.

parked_extents: Mutex<ParkedExtents>

File mode: entries whose extent class the store refused, set aside so that passes stop revisiting them. A pass returns a class’s entries to the front of extent_queue once the store can place that class.

parked_classes: AtomicU64

class_bit of every class with parked entries, so a pass skips the parked lock when nothing is parked.

file_release: Mutex<FileRelease>

File mode: the schedule on which spill threads release warm file store slots. See Pool::set_file_release.

queues: [Mutex<VecDeque<Weak<ChunkMeta>>>; 4]

Second-chance FIFOs of eviction candidates, one per depth band; a chunk joins the band of its ChunkHints depth at insert and again on re-admission. Entries for freed chunks go stale in place and are dropped by PoolInner::prune_queues.

Two scanners walk them with different obligations, both visiting the deepest band first. Budget enforcement is the one that ages chunks: it spends the touched bit (second chance) and drops entries it evicts. Eager backing (PoolInner::back_one) rotates visited entries to the back but never spends a touched bit, so a backing pass shuffles FIFO order without aging any chunk toward eviction.

extent_queue: Mutex<VecDeque<ExtentEntry>>

FIFO of chunks whose extents are resident, oldest first — the RSS-target enforcement’s victim queue. Entries go stale when an extent pages out, is dropped, or its chunk dies; visits drop them, and PoolInner::prune_extent_queue compacts dead-chunk entries that under-cap operation never visits.

live_chunks: AtomicU64

Number of live size-classed chunks (whatever their residency), which is the number of non-stale queue entries across all bands; PoolInner::prune_queues compacts the queues against it.

extent_residents: AtomicU64

Number of live chunks whose extent is currently resident, including unreclaimable extents that deliberately hold no extent_queue entry (heap-backed and retry-capped ones). It therefore upper-bounds the queue’s non-stale entries, and PoolInner::prune_extent_queue’s compaction threshold is conservative by the unreclaimable count.

enforcing: Mutex<()>

Single-flight claim for budget enforcement.

enforce_pending: AtomicBool

Set by an insert turned away from enforcing. The holder re-runs its pass while it is set, so a caller turned away after the holder’s final counter read still has its bytes enforced rather than dropped.

counters: Counters

spill: Spill

Implementations

impl PoolInner

fn queue(&self, band: usize) -> MutexGuard<'_, VecDeque<Weak<ChunkMeta>>>

Locks the eviction queue of one depth band.

fn extent_queue(&self) -> MutexGuard<'_, VecDeque<ExtentEntry>>

Locks the resident-extent queue.

fn parked_extents(&self) -> MutexGuard<'_, ParkedExtents>

Locks the parked resident-extent entries.

fn spill_queue(&self) -> MutexGuard<'_, VecDeque<Arc<ChunkMeta>>>

Locks the spill hand-off queue.

fn region_of(&self, meta: &ChunkMeta) -> &Region

The region behind a slotted chunk’s size class.

unsafe fn slot_data(&self, meta: &ChunkMeta, slot: u32) -> &[u64]

Borrows the payload of a slotted chunk.

Safety

slot must be meta’s slot, its contents must be initialized (they are from insert onward), and nothing may write the slot while the borrow lives.

fn commit_extent( &self, meta: &Arc<ChunkMeta>, state: &mut ChunkState, extent: Extent, )

Records a freshly written extent under the chunk’s state lock: the compressed-bytes counter, the compressed-tier accounting, and the state’s extent field.

fn prune_queues(&self)

Drops queue entries whose chunk has been freed, detected by their Weak no longer holding a live chunk. Each band compacts only when its stale entries outnumber all live chunks (plus a small floor), so the cost amortizes to a constant per insert and the total queue length stays proportional to the number of live slotted chunks even when the pool never comes under budget pressure.

fn reserve_insert(&self, len_bytes: usize) -> bool

Reserve insertion bytes before populating a slot. Enforcement can lag by insert_slack of the budget, or one payload for small budgets. Read admissions use the budget itself and cannot consume this slack.

fn enforce_budget(&self)

fn evictable_bytes(&self) -> u64

Bytes budget enforcement can actually reclaim: resident bytes minus heap-backed (oversize and class-exhaustion) chunks, which hold no slot and can never be evicted. Enforcing against raw resident bytes would, once unevictable bytes alone exceed the budget, compress every slotted chunk on arrival forever.

fn enforce_budget_inner(&self)

fn enforce_budget_band(&self, band: usize, quota: &mut u64)

fn evict_locked(&self, meta: &Arc<ChunkMeta>, state: &mut ChunkState)

fn spill_eligible(&self) -> bool

Whether the next eviction should be handed to spill threads: enabled, and the queue is below the backpressure bound (beyond it, callers evict inline rather than growing an unbounded queue of still-resident chunks).

fn spill_schedule(&self, meta: Arc<ChunkMeta>)

Hands a WriteInFlight chunk to the spill threads.

fn spill_worker(self: Arc<Self>)

Spill-thread main loop. The thread owns an Arc<PoolInner>, so the pool (a process-wide singleton in production) lives as long as its threads. Queued (budget-driven) evictions take priority; with eager backing enabled, idle threads compress unbacked chunks to BackedResident instead of parking, and park with a timeout once everything reachable is backed.

fn spill_trim(&self, hinted_retry: &mut Instant)

The spill loop’s compressed-tier trim. Tier-2 pageouts ride the spill threads: every pass through the loop (job completion, condvar wakeup, park timeout) trims the compressed tier if needed, at a single atomic load when under cap. While the file store’s full hint is set, a pass runs only once hinted_retry has passed, which then moves one SPILL_PARK_TIMEOUT ahead.

fn spill_file_upkeep(&self)

Re-derives the file store’s capacity and releases warm file store slots, each when due. The first spill thread to see one due claims its interval.

fn file_release(&self) -> MutexGuard<'_, FileRelease>

fn back_one(&self) -> bool

Eagerly compresses one unbacked chunk from the eviction queues into BackedResident, returning whether a chunk was backed — false means nothing was actionable (queues empty, or the bounded scans found only already-backed, in-flight, contended, or stale entries) and the caller should park rather than rescan. Bands are visited deepest first, mirroring eviction order so the chunks evicted first are the ones whose backing is already pre-paid.

fn back_one_from(&self, band: usize) -> bool

One bounded backing scan over a single band’s queue. Non-actionable entries are requeued or dropped per the same rules budget enforcement uses, except that the second-chance touched bit is left alone — backing is not an eviction and must not consume a chunk’s reprieve.

fn spill_process(&self, meta: &Arc<ChunkMeta>, kind: SpillKind)

Performs (or cancels) one scheduled compression. Runs on spill threads only. Lock discipline: the chunk lock is held only to validate and to commit, never across the compression or the pageout reclaim, which are the multi-millisecond costs this path exists to keep off budget-enforcing threads.

fn release_resident(&self, meta: &ChunkMeta, state: &mut ChunkState)

Releases whichever resident form the chunk holds, its slot or its typed body, under PoolInner::release_slot’s precondition.

fn release_slot(&self, meta: &ChunkMeta, state: &mut ChunkState)

Releases state’s slot — slot returned to the region free list, physical pages discarded unless the slot joins the bounded warm pool — and decrements resident bytes. Releasing pages beyond the warm pool is what keeps RSS aligned with the resident_bytes gauge the budget enforcer trusts; the warm pool relaxes that alignment by an explicit, bounded amount (warm_bytes, capped at a fraction of the budget) so slot reuse faults no pages and skips the kernel’s page zeroing.

Precondition: the caller holds the chunk’s state lock, and no reference into the slot exists — copy-out reads borrow the slot only under that same lock, and a WriteInFlight chunk’s unlocked compression read belongs to the spill thread, which is the only caller that releases the slot in that state. This is what makes the dontneed below sound, and what makes keeping a warm slot’s stale contents safe: the slot’s next occupant fully overwrites every byte it reads, satisfying the contents-undefined contract either way.

fn warm_cap(&self) -> u64

The warm pool’s byte ceiling: an eighth of the budget, clamped at an absolute maximum. The fraction sizes fault amortization at small budgets; the clamp keeps large budgets from parking gigabytes of idle warm slots no fault rate could justify.

fn trim_warm_pool(&self)

Cools warm free slots until warm_bytes falls to the warm cap. A budget shrink lowers the cap, and warm capacity is checked only when a slot is freed, so without this pass slots parked under the old cap would hold their pages until same-class reuse happened to drain them, exactly when the shrink wanted the memory back.

fn try_keep_warm(&self, class_size: usize) -> bool

Claims warm-pool capacity for a slot of class_size bytes, returning whether the slot may keep its pages. The RSS overshoot the warm pool introduces is bounded by PoolInner::warm_cap and visible as the warm_bytes stat.

fn try_alloc_slot(&self, class: usize, len_bytes: usize) -> Option<u32>

Allocates a slot in class for a payload of len_bytes with warm-pool accounting (a warm allocation is counted as a reuse and trimmed to the payload), or None when the class has no free slot.

fn trim_slot_tail(&self, class: usize, slot: u32, len_bytes: usize)

Releases a slot’s pages beyond the first len_bytes (rounded up to a page), so a slot reused for a smaller payload does not keep its prior occupant’s tail pages resident with no bytes in the ledger to answer for them.

Precondition: the caller exclusively owns the slot (freshly allocated, or taken from a victim under the victim’s state lock) with no reference into it.

fn alloc_slot(&self, class: usize, len_bytes: usize) -> Option<u32>

Allocates a slot in class for an insert: as PoolInner::try_alloc_slot, with an exhausted class counted as a heap fallback for a len_bytes payload (warned about once). None means the caller must degrade to the heap.

fn admit_slot(&self, meta: &ChunkMeta) -> Option<u32>

Acquires a slot for re-admitting an evicted chunk, from free budget headroom or by stealing a clean backed victim’s slot, never by evicting or compressing anything. None counts a denied admission. On success the admitted chunk’s resident-bytes accounting and the admission counter are settled, and the caller (who holds the chunk’s state lock) owns the slot: its contents are unspecified (fresh, warm, or the victim’s stale bytes) and must be fully overwritten.

fn steal_clean_victim( &self, class: usize, admitted_len_bytes: usize, ) -> Option<u32>

Takes the slot of a clean victim in class for an admitted payload of admitted_len_bytes: a BackedResident chunk with a clear touched bit, whose extent already duplicates its slot, so the victim transitions to Evicted with zero I/O, its extent intact, and its queue entry dropped. The returned slot keeps its physical pages (no dontneed, no free-list round trip); they hold the victim’s stale bytes. None when the bounded scan finds no such victim, or none whose growth the budget can absorb.

The caller holds its own chunk’s state lock. The scan follows the enforcement discipline (deepest band first, queue guard dropped before any chunk lock, victims only ever try_locked), which is what keeps the chunk-lock-while-probing-chunk-lock window deadlock-free: two admitters stealing toward each other both fail the try_lock and skip. Unlike enforcement, the scan rotates unsuitable entries (touched, wrong class, unbacked) to the back without spending touched bits, shuffling FIFO order the way the backing scan does.

fn compressed_cap(&self) -> u64

Capacity of the compressed-resident tier: the RSS target’s headroom above the slot budget, the insertion slack currently in use, and the warm cap. With no target set the tier has zero capacity, so extents page out as soon as they are written.

fn compressed_tier_above(&self, over: u64) -> bool

Whether the compressed tier holds more than over bytes, counted as enforcement counts it for the store.

fn insert_slack_in_use(&self, budget: u64) -> u64

Slot bytes reserved above budget by insertions that outran enforcement, at most insert_slack. Oversize payloads live on the heap and are excluded, as in PoolInner::reserve_insert.

fn note_extent_resident( &self, meta: &Arc<ChunkMeta>, extent_alloc: usize, reclaimable: bool, )

Counts a newly resident extent (written, or revived by a read) against the compressed tier. A reclaimable extent additionally enqueues its chunk for RSS-target enforcement; an unreclaimable one (a heap-fallback extent, which is never advised out) counts against the unreclaimable gauge instead and stays out of the queue, so enforcement never walks entries it cannot act on. Callers hold the chunk’s state lock with the extent present and resident, and follow up with PoolInner::enforce_compressed_cap once the lock is released.

Invariant: extent_resident_bytes equals the sum of alloc_size over live chunks’ extents whose is_resident() is true, and extent_residents counts those extents; extent_unreclaimable_bytes is the subset whose pageout_capped() is true. A freed chunk’s Demoting extent stays counted until its demoter’s commit releases it. This method, PoolInner::note_extent_reclaimable, PoolInner::note_extent_released, the pageout arms in PoolInner::enforce_compressed_cap, and the demotion commit in PoolInner::demote are the only adjusters. Every flag flip pairs with one of them under the chunk’s state lock.

Likewise extent_file_bytes equals the sum of file_bytes() over live chunks’ extents. The demotion commit adds to it and PoolInner::release_extent subtracts, each under the chunk’s state lock.

fn note_extent_reclaimable(&self, meta: &Arc<ChunkMeta>, extent_alloc: usize)

Returns a retry-capped resident extent to the reclaimable set after a read restored its pageout budget: uncounts it from the unreclaimable gauge and re-enqueues its chunk for RSS-target enforcement. The caller holds the chunk’s state lock with the extent present, resident, and no longer pageout_capped().

fn enqueue_extent(&self, meta: &Arc<ChunkMeta>, extent_alloc: usize)

Enqueues a reclaimable resident extent for RSS-target enforcement.

fn note_extent_released(&self, extent: &Extent)

Uncounts a resident extent that is being dropped (chunk freed or degraded). Its queue entry goes stale and is dropped on visit or by PoolInner::prune_extent_queue.

fn release_extent(&self, state: &mut ChunkState)

Drops a freed chunk’s extent with its accounting, under the chunk’s state lock. A Demoting extent stays in place, per Home::Demoting.

fn prune_extent_queue(&self)

Drops extent-queue entries whose chunk has been freed, mirroring PoolInner::prune_queues: compact only when the queue outgrows all live resident extents (plus a small floor), so the cost amortizes to a constant per push. Enforcement drops stale entries too, but only while the tier is over capacity. A pool that stays under its compressed cap would otherwise accumulate an entry (and a pin on the dead chunk’s allocation) per freed extent forever.

fn park_extent(&self, class: usize, entry: ExtentEntry)

Sets aside entry, whose extent class class the store refused, until a pass finds the class placeable. A class beyond the bit mask is requeued at the back instead.

fn unpark_placeable(&self, store: &FileStore)

Returns the parked entries of every class store can place now to the front of the extent queue, where they keep their age order ahead of the younger entries queued since.

fn enforce_or_defer_compressed_cap(&self)

Routes compressed-cap enforcement off latency-sensitive threads: with spill threads spawned, wakes one to perform the pageouts (MADV_PAGEOUT is synchronous reclaim, bounded per extent but not free at chunk rates). Without them, enforces inline unless the file store’s full hint is set. The test is for thread existence, not spill.enabled: spawned threads trim the tier in their loop even with eviction hand-off disabled.

Deferral makes the target eventually-enforced with bounded lag, and the backstop below bounds the lag: a caller finding the reclaimable tier at double its capacity enforces inline, so sustained creation cannot outrun trimming by more than one capacity’s worth. In file mode the bound does not hold while the store’s full hint is set or its writes are disabled. The backstop then defers to the spill threads, and the tier grows past the bound by whatever the store cannot take.

fn reclaimable_resident(&self) -> u64

Bytes of resident extents RSS-target enforcement can push out.

fn enforce_compressed_cap(&self, pass: Pass)

Pages out the oldest resident extents until the compressed tier falls to its capacity. The compression is already paid and the device write is the kernel’s async writeback, so each pageout is one bounded madvise plus a page-table observation; spill threads run this between jobs, and other threads only when no spill threads exist (see PoolInner::enforce_or_defer_compressed_cap). Not single-flighted: concurrent passes pop disjoint victims. Visits are bounded by the queue’s length at entry; stale entries (extent paged out, dropped, or chunk dead) are dropped. Incomplete extents are requeued with their accounting intact until their retry budget runs out, at which point they leave the queue with their bytes on the unreclaimable gauge, so the tier may settle above its capacity by the bytes the kernel declined to reclaim without enforcement re-walking them.

In file mode each victim is demoted to the file store instead, per PoolInner::demote. A victim whose class the store cannot place moves to the back of the queue, and a store that can place no class or has writes disabled ends the pass. An inline file-mode pass with spill threads present stops at the backstop_threshold, and a background file-mode pass stops per DEMOTION_PASS_BYTES. An inline file-mode pass with spill threads present also stops after DEMOTION_PASS_BYTES.

fn demote( &self, store: &Arc<FileStore>, meta: &Arc<ChunkMeta>, state: MutexGuard<'_, ChunkState>, entry: ExtentEntry, pass: Pass, ) -> Demotion

Demotes the extent of the chunk behind state to store, the file mode counterpart of the pageout arms of PoolInner::enforce_compressed_cap. entry is the chunk’s popped queue entry, which this function requeues unless the entry is stale or the extent left the arena.

The write runs without the chunk lock, with the extent Demoting (see Home::Demoting in extent.rs).

fn refuse( &self, store: &FileStore, class: usize, entry: ExtentEntry, ) -> Demotion

Parks or requeues the entry of an extent whose class the store could not take.

fn spill_handoff(&self, meta: &Arc<ChunkMeta>, state: &mut ChunkState) -> bool

If the chunk is a live UnbackedResident holding a slot and the spill threads have capacity, transitions it to WriteInFlight and hands it to them, returning true. The hand-off happens under the held state lock; the spill thread blocks on that lock only after this call returns and the caller releases it.

Trait Implementations

impl Debug for PoolInner

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.