Struct PoolInner 

struct PoolInner {
    budget_bytes: AtomicU64,
    rss_target_bytes: AtomicU64,
    regions: Vec<Region>,
    extent_arena: Arc<ExtentArena>,
    queues: [Mutex<VecDeque<Weak<ChunkMeta>>>; 4],
    extent_queue: Mutex<VecDeque<Weak<ChunkMeta>>>,
    live_chunks: AtomicU64,
    extent_residents: AtomicU64,
    enforcing: Mutex<()>,
    enforce_pending: AtomicBool,
    counters: Counters,
    spill: Spill,
}

Available on Unix and crate feature pool only.

The shared state behind every Pool handle: the budget and RSS ledgers, the size-class slot regions and the extent arena, the eviction and backing queues, and the spill-thread hand-off. One per process in practice; Pool clones and chunk handles share it through an Arc, so it lives until the last handle and spill thread release it.

Lock order: a chunk’s state mutex may be held while taking any of the leaf locks — the eviction queue, the extent_queue, the spill queue, and the region slot allocators — but never the reverse. The enforcement and backing scans additionally drop the queue guard before trying a chunk’s state lock (and only ever try_lock it), so no path holds a queue lock while waiting on chunk state. The admitting read’s victim steal is the one place a chunk’s state lock is held while probing another chunk’s, and the victim is only ever try_locked, so two admitters stealing toward each other skip instead of deadlocking. Reads copy out under the chunk’s state lock — the same lock eviction takes — so there is no reader-side count and no reader the evictor must account for.

Fields

budget_bytes: AtomicU64

Resident-bytes target, enforced against evictable bytes (resident minus heap-backed, which no eviction can reclaim). Atomic so a running pool can be retuned in place (operator-driven budget changes) without orphaning live handles, which share this value through their Arc<PoolInner>.

rss_target_bytes: AtomicU64

Ceiling on the pool’s total RSS: slots (the budget) plus warm free slots plus compressed-resident extents. The compressed tier’s capacity derives as max(0, rss_target - budget - warm cap); zero (the default) collapses the tier, paging every extent out as soon as it is written.

regions: Vec<Region>

One region per entry of SIZE_CLASSES, same order.

extent_arena: Arc<ExtentArena>

The arena backing extents. Shared with every live SwapExtent, whose drop returns its slot.

queues: [Mutex<VecDeque<Weak<ChunkMeta>>>; 4]

Second-chance FIFOs of eviction candidates, one per depth band; a chunk joins the band of its ChunkHints depth at insert and again on re-admission. Entries for freed chunks go stale in place and are dropped by PoolInner::prune_queues.

Two scanners walk them with different obligations, both visiting the deepest band first. Budget enforcement is the one that ages chunks: it spends the touched bit (second chance) and drops entries it evicts. Eager backing (PoolInner::back_one) rotates visited entries to the back but never spends a touched bit, so a backing pass shuffles FIFO order without aging any chunk toward eviction.

extent_queue: Mutex<VecDeque<Weak<ChunkMeta>>>

FIFO of chunks whose extents are resident, oldest first — the RSS-target enforcement’s victim queue. Entries go stale when an extent pages out, is dropped, or its chunk dies; visits drop them, and PoolInner::prune_extent_queue compacts dead-chunk entries that under-cap operation never visits.

live_chunks: AtomicU64

Number of live size-classed chunks (whatever their residency), which is the number of non-stale queue entries across all bands; PoolInner::prune_queues compacts the queues against it.

extent_residents: AtomicU64

Number of live chunks whose extent is currently resident, including unreclaimable extents that deliberately hold no extent_queue entry (heap-backed and retry-capped ones). It therefore upper-bounds the queue’s non-stale entries, and PoolInner::prune_extent_queue’s compaction threshold is conservative by the unreclaimable count.

enforcing: Mutex<()>

Single-flight claim for budget enforcement.

enforce_pending: AtomicBool

Set by an insert turned away from enforcing. The holder re-runs its pass while it is set, so a caller turned away after the holder’s final counter read still has its bytes enforced rather than dropped.

counters: Counters

spill: Spill

Implementations

impl PoolInner

fn queue(&self, band: usize) -> MutexGuard<'_, VecDeque<Weak<ChunkMeta>>>

Locks the eviction queue of one depth band.

fn extent_queue(&self) -> MutexGuard<'_, VecDeque<Weak<ChunkMeta>>>

Locks the resident-extent queue.

fn spill_queue(&self) -> MutexGuard<'_, VecDeque<Arc<ChunkMeta>>>

Locks the spill hand-off queue.

fn region_of(&self, meta: &ChunkMeta) -> &Region

The region behind a slotted chunk’s size class.

unsafe fn slot_data(&self, meta: &ChunkMeta, slot: u32) -> &[u64]

Borrows the payload of a slotted chunk.

Safety

slot must be meta’s slot, its contents must be initialized (they are from insert onward), and nothing may write the slot while the borrow lives.

fn commit_extent( &self, meta: &Arc<ChunkMeta>, state: &mut ChunkState, extent: SwapExtent, )

Records a freshly written extent under the chunk’s state lock: the compressed-bytes counter, the compressed-tier accounting, and the state’s extent field.

fn prune_queues(&self)

Drops queue entries whose chunk has been freed, detected by their Weak no longer holding a live chunk. Each band compacts only when its stale entries outnumber all live chunks (plus a small floor), so the cost amortizes to a constant per insert and the total queue length stays proportional to the number of live slotted chunks even when the pool never comes under budget pressure.

fn enforce_budget(&self)

fn evictable_bytes(&self) -> u64

Bytes budget enforcement can actually reclaim: resident bytes minus heap-backed (oversize and class-exhaustion) chunks, which hold no slot and can never be evicted. Enforcing against raw resident bytes would, once unevictable bytes alone exceed the budget, compress every slotted chunk on arrival forever.

fn enforce_budget_inner(&self)

fn enforce_budget_band(&self, band: usize)

fn evict_locked(&self, meta: &Arc<ChunkMeta>, state: &mut ChunkState)

fn spill_eligible(&self) -> bool

Whether the next eviction should be handed to spill threads: enabled, and the queue is below the backpressure bound (beyond it, callers evict inline rather than growing an unbounded queue of still-resident chunks).

fn spill_schedule(&self, meta: Arc<ChunkMeta>)

Hands a WriteInFlight chunk to the spill threads.

fn spill_worker(self: Arc<Self>)

Spill-thread main loop. The thread owns an Arc<PoolInner>, so the pool (a process-wide singleton in production) lives as long as its threads. Queued (budget-driven) evictions take priority; with eager backing enabled, idle threads compress unbacked chunks to BackedResident instead of parking, and park with a timeout once everything reachable is backed.

fn back_one(&self) -> bool

Eagerly compresses one unbacked chunk from the eviction queues into BackedResident, returning whether a chunk was backed — false means nothing was actionable (queues empty, or the bounded scans found only already-backed, in-flight, contended, or stale entries) and the caller should park rather than rescan. Bands are visited deepest first, mirroring eviction order so the chunks evicted first are the ones whose backing is already pre-paid.

fn back_one_from(&self, band: usize) -> bool

One bounded backing scan over a single band’s queue. Non-actionable entries are requeued or dropped per the same rules budget enforcement uses, except that the second-chance touched bit is left alone — backing is not an eviction and must not consume a chunk’s reprieve.

fn spill_process(&self, meta: &Arc<ChunkMeta>, kind: SpillKind)

Performs (or cancels) one scheduled compression. Lock discipline: the chunk lock is held only to validate and to commit — never across the compression or the pageout reclaim, which are the multi-millisecond costs this path exists to keep off budget-enforcing threads.

fn release_slot(&self, meta: &ChunkMeta, state: &mut ChunkState)

Releases state’s slot — slot returned to the region free list, physical pages discarded unless the slot joins the bounded warm pool — and decrements resident bytes. Releasing pages beyond the warm pool is what keeps RSS aligned with the resident_bytes gauge the budget enforcer trusts; the warm pool relaxes that alignment by an explicit, bounded amount (warm_bytes, capped at a fraction of the budget) so slot reuse faults no pages and skips the kernel’s page zeroing.

Precondition: the caller holds the chunk’s state lock, and no reference into the slot exists — copy-out reads borrow the slot only under that same lock, and a WriteInFlight chunk’s unlocked compression read belongs to the spill thread, which is the only caller that releases the slot in that state. This is what makes the dontneed below sound, and what makes keeping a warm slot’s stale contents safe: the slot’s next occupant fully overwrites every byte it reads, satisfying the contents-undefined contract either way.

fn warm_cap(&self) -> u64

The warm pool’s byte ceiling: an eighth of the budget, clamped at an absolute maximum. The fraction sizes fault amortization at small budgets; the clamp keeps large budgets from parking gigabytes of idle warm slots no fault rate could justify.

fn trim_warm_pool(&self)

Cools warm free slots until warm_bytes falls to the warm cap. A budget shrink lowers the cap, and warm capacity is checked only when a slot is freed, so without this pass slots parked under the old cap would hold their pages until same-class reuse happened to drain them, exactly when the shrink wanted the memory back.

fn try_keep_warm(&self, class_size: usize) -> bool

Claims warm-pool capacity for a slot of class_size bytes, returning whether the slot may keep its pages. The RSS overshoot the warm pool introduces is bounded by PoolInner::warm_cap and visible as the warm_bytes stat.

fn try_alloc_slot(&self, class: usize, len_bytes: usize) -> Option<u32>

Allocates a slot in class for a payload of len_bytes with warm-pool accounting (a warm allocation is counted as a reuse and trimmed to the payload), or None when the class has no free slot.

fn trim_slot_tail(&self, class: usize, slot: u32, len_bytes: usize)

Releases a slot’s pages beyond the first len_bytes (rounded up to a page), so a slot reused for a smaller payload does not keep its prior occupant’s tail pages resident with no bytes in the ledger to answer for them.

Precondition: the caller exclusively owns the slot (freshly allocated, or taken from a victim under the victim’s state lock) with no reference into it.

fn alloc_slot(&self, class: usize, len_bytes: usize) -> Option<u32>

Allocates a slot in class for an insert: as PoolInner::try_alloc_slot, with an exhausted class counted as a heap fallback for a len_bytes payload (warned about once). None means the caller must degrade to the heap.

fn admit_slot(&self, meta: &ChunkMeta) -> Option<u32>

Acquires a slot for re-admitting an evicted chunk, from free budget headroom or by stealing a clean backed victim’s slot, never by evicting or compressing anything. None counts a denied admission. On success the admitted chunk’s resident-bytes accounting and the admission counter are settled, and the caller (who holds the chunk’s state lock) owns the slot: its contents are unspecified (fresh, warm, or the victim’s stale bytes) and must be fully overwritten.

fn steal_clean_victim( &self, class: usize, admitted_len_bytes: usize, ) -> Option<u32>

Takes the slot of a clean victim in class for an admitted payload of admitted_len_bytes: a BackedResident chunk with a clear touched bit, whose extent already duplicates its slot, so the victim transitions to Evicted with zero I/O, its extent intact, and its queue entry dropped. The returned slot keeps its physical pages (no dontneed, no free-list round trip). They hold the victim’s stale bytes. The ledger is settled inside the steal: the victim’s bytes leave and the admitted payload’s enter in one step, and a steal that grows resident bytes must fit the budget like any other admission (a shrinking steal always may proceed). None when the bounded scan finds no such victim, or none whose growth the budget can absorb.

The caller holds its own chunk’s state lock. The scan follows the enforcement discipline (deepest band first, queue guard dropped before any chunk lock, victims only ever try_locked), which is what keeps the chunk-lock-while-probing-chunk-lock window deadlock-free: two admitters stealing toward each other both fail the try_lock and skip. Unlike enforcement, the scan rotates unsuitable entries (touched, wrong class, unbacked) to the back without spending touched bits, shuffling FIFO order the way the backing scan does.

fn compressed_cap(&self) -> u64

Capacity of the compressed-resident tier: the RSS target’s headroom above the slot budget and the warm cap. With no target set the tier has zero capacity, so extents page out as soon as they are written.

fn note_extent_resident( &self, meta: &Arc<ChunkMeta>, extent_alloc: usize, reclaimable: bool, )

Counts a newly resident extent (written, or revived by a read) against the compressed tier. A reclaimable extent additionally enqueues its chunk for RSS-target enforcement; an unreclaimable one (a heap-fallback extent, which is never advised out) counts against the unreclaimable gauge instead and stays out of the queue, so enforcement never walks entries it cannot act on. Callers hold the chunk’s state lock with the extent present and resident, and follow up with PoolInner::enforce_compressed_cap once the lock is released.

Invariant: extent_resident_bytes equals the sum of alloc_size over live chunks’ extents whose is_resident() is true, and extent_residents counts those extents; extent_unreclaimable_bytes is the subset whose pageout_capped() is true. This method, PoolInner::note_extent_reclaimable, PoolInner::note_extent_released, and the pageout arms in PoolInner::enforce_compressed_cap are the only adjusters; every flag flip pairs with one of them under the chunk’s state lock.

fn note_extent_reclaimable(&self, meta: &Arc<ChunkMeta>, extent_alloc: usize)

Returns a retry-capped resident extent to the reclaimable set after a read restored its pageout budget: uncounts it from the unreclaimable gauge and re-enqueues its chunk for RSS-target enforcement. The caller holds the chunk’s state lock with the extent present, resident, and no longer pageout_capped().

fn note_extent_released(&self, extent: &SwapExtent)

Uncounts a resident extent that is being dropped (chunk freed or degraded). Its queue entry goes stale and is dropped on visit or by PoolInner::prune_extent_queue.

fn prune_extent_queue(&self)

Drops extent-queue entries whose chunk has been freed, mirroring PoolInner::prune_queues: compact only when the queue outgrows all live resident extents (plus a small floor), so the cost amortizes to a constant per push. Enforcement drops stale entries too, but only while the tier is over capacity. A pool that stays under its compressed cap would otherwise accumulate an entry (and a pin on the dead chunk’s allocation) per freed extent forever.

fn enforce_or_defer_compressed_cap(&self)

Routes compressed-cap enforcement off latency-sensitive threads: with spill threads running, wakes one to perform the pageouts (MADV_PAGEOUT is synchronous reclaim — page-table walks, TLB shootdowns, writeback submission — bounded per compressed extent but not free at chunk rates); without them, enforces inline as the only option.

The routing rule across the two ceilings: budget pressure goes through PoolInner::enforce_budget, single-flighted because concurrent passes would convoy on redundant compression scans; tier pressure goes through this router, and the enforcement itself is deliberately not single-flighted, since concurrent passes pop disjoint victims and each visit is microseconds. Spill threads call PoolInner::enforce_compressed_cap directly (they are the deferral target), and Pool::set_rss_target enforces a shrink inline so config changes land synchronously, mirroring set_budget.

Deferral makes the target eventually-enforced with bounded lag (a notify with every spill thread mid-job is absorbed; the next loop pass catches up). The backstop below turns that into a bound by construction: a caller finding the tier at double its capacity enforces inline regardless, so sustained creation can never outrun trimming by more than one capacity’s worth.

Deferral tests for thread existence alone, not spill.enabled: spawned threads trim the tier in their loop for as long as they live, even with eviction hand-off disabled, so they remain the better home for the pageouts.

fn enforce_compressed_cap(&self)

Pages out the oldest resident extents until the compressed tier falls to its capacity. The compression is already paid and the device write is the kernel’s async writeback, so each pageout is one bounded madvise plus a page-table observation; spill threads run this between jobs, and other threads only when no spill threads exist (see PoolInner::enforce_or_defer_compressed_cap). Not single-flighted: concurrent passes pop disjoint victims. Visits are bounded by the queue’s length at entry; stale entries (extent paged out, dropped, or chunk dead) are dropped. Incomplete extents are requeued with their accounting intact until their retry budget runs out, at which point they leave the queue with their bytes on the unreclaimable gauge, so the tier may settle above its capacity by the bytes the kernel declined to reclaim without enforcement re-walking them.

fn spill_handoff(&self, meta: &Arc<ChunkMeta>, state: &mut ChunkState) -> bool

If the chunk is a live UnbackedResident holding a slot and the spill threads have capacity, transitions it to WriteInFlight and hands it to them, returning true. The hand-off happens under the held state lock; the spill thread blocks on that lock only after this call returns and the caller releases it.

Trait Implementations

impl Debug for PoolInner

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.