Struct mz_persist::indexed::columnar::ColumnarRecords

pub struct ColumnarRecords { /* private fields */ }

A set of ((Key, Val), Time, Diff) records stored in a columnar representation.

Note that the data are unsorted, and unconsolidated (so there may be multiple instances of the same ((Key, Val), Time), and some Diffs might be zero, or add up to zero).

Both Time and Diff are presented externally to persist users as a type parameter that implements mz_persist_types::Codec64. Our columnar format intentionally stores them both as i64 columns (as opposed to something like a fixed width binary column) because this allows us additional compression options.

Also note that we intentionally use an i64 over a u64 for Time. Over the range [0, i64::MAX], the bytes are the same and we’ve talked at various times about changing Time in mz to an i64. Both millis since unix epoch and nanos since unix epoch easily fit into this range (the latter until some time after year 2200). Using a i64 might be a pessimization for a non-realtime mz source with u64 timestamps in the range (i64::MAX, u64::MAX], but realtime sources are overwhelmingly the common case.

The i’th key’s data is stored in key_data[key_offsets[i]..key_offsets[i+1]]. Similarly for val.

Invariants:

• len < usize::MAX (so len+1 can fit in a usize)
• key_offsets.len() * BYTES_PER_KEY_VAL_OFFSET + key_data.len() <= KEY_VAL_DATA_MAX_LEN
• key_offsets.len() == len + 1
• key_offsets are non-decreasing
• Each key_offset is <= key_data.len()
• key_offsets.first().unwrap() == 0
• key_offsets.last().unwrap() == key_data.len()
• val_offsets.len() * BYTES_PER_KEY_VAL_OFFSET + val_data.len() <= KEY_VAL_DATA_MAX_LEN
• val_offsets.len() == len + 1
• val_offsets are non-decreasing
• Each val_offset is <= val_data.len()
• val_offsets.first().unwrap() == 0
• val_offsets.last().unwrap() == val_data.len()
• timestamps.len() == len
• diffs.len() == len

Implementations

impl ColumnarRecords

pub fn len(&self) -> usize

The number of (potentially duplicated) ((Key, Val), Time, i64) records stored in Self.

pub fn goodbytes(&self) -> usize

The number of logical bytes in the represented data, excluding offsets and lengths.

pub fn get<'a>( &'a self, idx: usize ) -> Option<((&'a [u8], &'a [u8]), [u8; 8], [u8; 8])>

Read the record at idx, if there is one.

Returns None if idx >= self.len().

pub fn iter<'a>(&'a self) -> ColumnarRecordsIter<'a>

Iterate through the records in Self.

impl ColumnarRecords

pub fn into_proto(&self) -> ProtoColumnarRecords

See RustType::into_proto.

pub fn from_proto( lgbytes: &ColumnarMetrics, proto: ProtoColumnarRecords ) -> Result<Self, TryFromProtoError>

See RustType::from_proto.

Trait Implementations

impl Clone for ColumnarRecords

fn clone(&self) -> ColumnarRecords

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ColumnarRecords

fn fmt(&self, fmt: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for ColumnarRecords

fn eq(&self, other: &ColumnarRecords) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for ColumnarRecords