pool only.Expand description
Size-class virtual-memory regions for the buffer pool.
One Region per size class, each a single anonymous mmap reservation.
The reservation is virtual; physical memory materializes on first write to
a slot. Slots are scoped to residency: eviction releases a slot’s physical
pages with dontneed and returns the slot index to the free list, so a
chunk holds a slot only from insert until its eviction, and the pool reads
slots strictly copy-out under the owning chunk’s state lock.
Two fault-amortization mechanisms soften the cost of cycling slots:
- Regions whose class is at least one huge page are aligned to the huge
page and advised
MADV_HUGEPAGE, so populating a large slot costs one fault instead of one per 4 KiB. - The free list is split into a warm side (pages kept resident; reuse faults nothing and skips the kernel’s page zeroing) and a cold side (pages released). The pool decides which side a freed slot joins, bounding total warm bytes as a fraction of its budget.
TODO: consider mlocking slot regions so kernel swap can never write
out pages the engine would discard or write better itself. Needs
RLIMIT_MEMLOCK tuning before it can be on by default.
All platform access goes through the sys seam: mapping, unmapping,
paging advice, and the page size. Under Miri the seam swaps to a
Rust-heap backing with advice as a contents-preserving no-op, so the
pool’s tests (including the unsafe slot borrows they exercise) run under
the interpreter.
Modules§
- sys 🔒
Non- miri - The platform seam: mapping, unmapping, paging advice, and the page size.
Structs§
- Region 🔒
- One anonymous virtual-memory reservation serving fixed-size slots of a single size class.
- Slot
Allocator 🔒 - Free-list-plus-bump slot allocator. A slot index returns to a free list whenever its chunk stops being resident — eviction and free alike. Warm slots keep their physical pages (reuse is fault-free); cold slots had theirs released. Never-allocated slots beyond the high-water mark are untouched virtual space and fault on first write like cold ones.
Constants§
- HUGE_
PAGE 🔒 - The transparent-huge-page size assumed for region alignment. Linux x86-64 and aarch64 (4 KiB base pages) both use 2 MiB; if a platform differs, the alignment is merely unhelpful, never wrong.
- SIZE_
CLASSES 🔒 - Chunk size classes in bytes, smallest first. The pool places each chunk in the smallest class that fits its payload.
Functions§
- align_
trim 🔒 - Splits an over-mapped range of
map_lenbytes ataddrinto(head, tail)trim amounts such that discardingheadbytes from the front andtailfrom the back leaves analign-aligned range of exactlylenbytes.Noneon address-space overflow or when the range cannot fit an alignedlenbytes. - aligned_
subrange 🔒 - The largest
page-aligned subrange of[addr, addr + len), as a(byte offset from addr, subrange length)pair, orNonewhen the range covers no whole page (including on address-space overflow).pagemust be a power of two. - dontdump 🔒
- Excludes the page-aligned subrange of
[ptr, ptr + len)from core dumps. Contents are preserved. No-op outside Linux. - dontneed 🔒 ⚠
- Releases the physical pages of the page-aligned subrange of
[ptr, ptr + len), keeping the virtual range mapped. - nohugepage 🔒
- Opts the page-aligned subrange of
[ptr, ptr + len)out of transparent huge pages, so reclaim advice over the range operates on base pages and never needs a folio split. Contents are preserved. No-op outside Linux. - nonresident 🔒
- Whether every page of the page-aligned subrange of
[ptr, ptr + len)has been unmapped from this process, per the pagemap present bits: the observation the pageout ledger trusts instead of the reclaim advice’s return value. A page unmapped to a swap entry counts as reclaimed even while its clean copy lingers in the kernel’s swap cache. Errs towardfalse(resident) when the observation is unavailable. In test builds the answer comes from thefake_residencyseam instead of the platform. - page_
size 🔒 - The system page size.
- pageout 🔒
- Hints the kernel to reclaim the page-aligned subrange of
[ptr, ptr + len)immediately, writing it to the swap device. Contents are preserved; this is a non-destructive hint. No-op outside Linux. - size_
class_ 🔒for - The smallest size class that fits a payload of
len_bytes, orNonewhen even the largest class is too small. A selected class always fits:SIZE_CLASSES[class] >= len_bytes, the bound the pool turns into slice lengths over slot memory (proved by the Kani harnesses). - willneed 🔒
- Hints the kernel to fault the page-aligned subrange of
[ptr, ptr + len)back in ahead of need: asynchronous swap-in, the swap-backed extent store’s readahead mechanism. Contents are preserved. No-op outside Linux.