Struct ColumnarRecords

pub struct ColumnarRecords {
    len: usize,
    key_data: BinaryArray,
    val_data: BinaryArray,
    timestamps: Int64Array,
    diffs: Int64Array,
}

A set of ((Key, Val), Time, Diff) records stored in a columnar representation.

Note that the data are unsorted, and unconsolidated (so there may be multiple instances of the same ((Key, Val), Time), and some Diffs might be zero, or add up to zero).

Both Time and Diff are presented externally to persist users as a type parameter that implements mz_persist_types::Codec64. Our columnar format intentionally stores them both as i64 columns (as opposed to something like a fixed width binary column) because this allows us additional compression options.

Also note that we intentionally use an i64 over a u64 for Time. Over the range [0, i64::MAX], the bytes are the same and we’ve talked at various times about changing Time in mz to an i64. Both millis since unix epoch and nanos since unix epoch easily fit into this range (the latter until some time after year 2200). Using a i64 might be a pessimization for a non-realtime mz source with u64 timestamps in the range (i64::MAX, u64::MAX], but realtime sources are overwhelmingly the common case.

Invariants:

• len <= u32::MAX (since we use arrow’s BinaryArray for our binary data)
• the length of all arrays should == len
• all entries should be non-null

Fields

len: usize

key_data: BinaryArray

val_data: BinaryArray

timestamps: Int64Array

diffs: Int64Array

Implementations

impl ColumnarRecords

pub fn new( key_data: BinaryArray, val_data: BinaryArray, timestamps: Int64Array, diffs: Int64Array, ) -> Self

Construct a columnar records instance from the given arrays, checking invariants.

pub fn len(&self) -> usize

The number of (potentially duplicated) ((Key, Val), Time, i64) records stored in Self.

pub fn keys(&self) -> &BinaryArray

The keys in this columnar records as an array.

pub fn vals(&self) -> &BinaryArray

The vals in this columnar records as an array.

pub fn timestamps(&self) -> &Int64Array

The timestamps in this columnar records as an array.

pub fn diffs(&self) -> &Int64Array

The diffs in this columnar records as an array.

pub fn goodbytes(&self) -> usize

The number of logical bytes in the represented data, excluding offsets and lengths.

pub fn get(&self, idx: usize) -> Option<((&[u8], &[u8]), [u8; 8], [u8; 8])>

Read the record at idx, if there is one.

Returns None if idx >= self.len().

pub fn iter( &self, ) -> impl Iterator<Item = ((&[u8], &[u8]), [u8; 8], [u8; 8])> + ExactSizeIterator

Iterate through the records in Self.

pub fn concat( records: &[ColumnarRecords], metrics: &ColumnarMetrics, ) -> ColumnarRecords

Concatenate the given records together, column-by-column.

impl ColumnarRecords

fn validate(&self) -> Result<(), String>

Trait Implementations

impl Clone for ColumnarRecords

fn clone(&self) -> ColumnarRecords

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ColumnarRecords

fn fmt(&self, fmt: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ColumnarRecords

fn default() -> Self

Returns the “default value” for a type.

impl PartialEq for ColumnarRecords

fn eq(&self, other: &ColumnarRecords) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for ColumnarRecords