Struct DictionaryCodec 

pub struct DictionaryCodec {
    encode: BTreeMap<Vec<u8>, u8>,
    pub decode: BytesMap,
    stats: (MisraGries<Vec<u8>>, [u64; 4]),
}

Per-column dictionary codec. Encodes column byte slices, replacing popular values with spare tags; decoding is performed by ColumnsIter reading the decode map directly.

Fields

encode: BTreeMap<Vec<u8>, u8>

decode: BytesMap

stats: (MisraGries<Vec<u8>>, [u64; 4])

Implementations

impl DictionaryCodec

pub(super) fn encode<'a, I>(&mut self, iter: I, output: &mut Vec<u8>)

where I: IntoIterator<Item = &'a [u8]>,

Encode a sequence of byte slices.

Encoding also records statistics about the structure of the input.

Decoding has no symmetric method here: a column’s bytes are decoded by ColumnsIter, which consults the decode map directly.

pub(super) fn new_from<'a>(stats: impl IntoIterator<Item = &'a Self>) -> Self

Construct a new encoder from supplied statistics.

impl DictionaryCodec

pub fn heap_size(&self, callback: &mut impl FnMut(usize, usize))

Visit contained allocations to determine their size and capacity.

BTreeMap exposes no capacity, so its node storage is approximated as one logical entry’s worth of bytes per element; the dominant terms (the owned key bytes and the decode map’s byte arena) are accounted exactly.

pub fn observe(&mut self, bytes: &[u8])

Record a single column value in this codec’s statistics without producing any encoded output.

Statistics come in two decoupled parts, with very different costs and purposes:

1. The tag bitmap (stats.1) records which first-byte values have been observed. It is cheap (four u64 ORs) and soundness critical: new_from’s dynamic-tag path only hands out tags that this bitmap reports as unused, so it must stay accurate for the entire life of the codec, including on the hot encode path.
2. The MisraGries summary (stats.0) tracks heavy hitters and only affects which values a future codec compresses, never correctness. It is the expensive part (a BTreeMap insert per column per row). We keep feeding it after install, on the hot encode path, on purpose: a later merge rebuilds the merged codec from these summaries via new_from. If we froze the summary at install time, then as the collection evolves — records cancel under consolidation, the popular set drifts — the codec could never reclaim slots for newly-popular values and would eventually be left compressing values that no longer occur, ceasing to compress the ones that do.

pub(super) fn new_safe(stats: Self) -> Self

Construct a codec using only structurally safe tags (>= SAFE_TAG_BASE). These tags never collide with datum first-bytes, so the codec can be installed without observing all data first.

Trait Implementations

impl Debug for DictionaryCodec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DictionaryCodec

fn default() -> DictionaryCodec

Returns the “default value” for a type.