Struct mz_persist::indexed::columnar::ColumnarRecords
source · pub struct ColumnarRecords { /* private fields */ }
Expand description
A set of ((Key, Val), Time, Diff) records stored in a columnar representation.
Note that the data are unsorted, and unconsolidated (so there may be multiple instances of the same ((Key, Val), Time), and some Diffs might be zero, or add up to zero).
Both Time and Diff are presented externally to persist users as a type parameter that implements mz_persist_types::Codec64. Our columnar format intentionally stores them both as i64 columns (as opposed to something like a fixed width binary column) because this allows us additional compression options.
Also note that we intentionally use an i64 over a u64 for Time. Over the
range [0, i64::MAX]
, the bytes are the same and we’ve talked at various
times about changing Time in mz to an i64. Both millis since unix epoch and
nanos since unix epoch easily fit into this range (the latter until some
time after year 2200). Using a i64 might be a pessimization for a
non-realtime mz source with u64 timestamps in the range (i64::MAX, u64::MAX]
, but realtime sources are overwhelmingly the common case.
The i’th key’s data is stored in
key_data[key_offsets[i]..key_offsets[i+1]]
. Similarly for val.
Invariants:
- len < usize::MAX (so len+1 can fit in a usize)
- key_offsets.len() * BYTES_PER_KEY_VAL_OFFSET + key_data.len() <= KEY_VAL_DATA_MAX_LEN
- key_offsets.len() == len + 1
- key_offsets are non-decreasing
- Each key_offset is <= key_data.len()
- key_offsets.first().unwrap() == 0
- key_offsets.last().unwrap() == key_data.len()
- val_offsets.len() * BYTES_PER_KEY_VAL_OFFSET + val_data.len() <= KEY_VAL_DATA_MAX_LEN
- val_offsets.len() == len + 1
- val_offsets are non-decreasing
- Each val_offset is <= val_data.len()
- val_offsets.first().unwrap() == 0
- val_offsets.last().unwrap() == val_data.len()
- timestamps.len() == len
- diffs.len() == len
Implementations§
source§impl ColumnarRecords
impl ColumnarRecords
sourcepub fn len(&self) -> usize
pub fn len(&self) -> usize
The number of (potentially duplicated) ((Key, Val), Time, i64) records stored in Self.
sourcepub fn goodbytes(&self) -> usize
pub fn goodbytes(&self) -> usize
The number of logical bytes in the represented data, excluding offsets and lengths.
sourcepub fn get<'a>(
&'a self,
idx: usize
) -> Option<((&'a [u8], &'a [u8]), [u8; 8], [u8; 8])>
pub fn get<'a>( &'a self, idx: usize ) -> Option<((&'a [u8], &'a [u8]), [u8; 8], [u8; 8])>
Read the record at idx
, if there is one.
Returns None if idx >= self.len()
.
sourcepub fn iter<'a>(&'a self) -> ColumnarRecordsIter<'a> ⓘ
pub fn iter<'a>(&'a self) -> ColumnarRecordsIter<'a> ⓘ
Iterate through the records in Self.
source§impl ColumnarRecords
impl ColumnarRecords
sourcepub fn into_proto(&self) -> ProtoColumnarRecords
pub fn into_proto(&self) -> ProtoColumnarRecords
See RustType::into_proto.
sourcepub fn from_proto(
lgbytes: &ColumnarMetrics,
proto: ProtoColumnarRecords
) -> Result<Self, TryFromProtoError>
pub fn from_proto( lgbytes: &ColumnarMetrics, proto: ProtoColumnarRecords ) -> Result<Self, TryFromProtoError>
See RustType::from_proto.
Trait Implementations§
source§impl Clone for ColumnarRecords
impl Clone for ColumnarRecords
source§fn clone(&self) -> ColumnarRecords
fn clone(&self) -> ColumnarRecords
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moresource§impl Debug for ColumnarRecords
impl Debug for ColumnarRecords
source§impl PartialEq for ColumnarRecords
impl PartialEq for ColumnarRecords
source§fn eq(&self, other: &ColumnarRecords) -> bool
fn eq(&self, other: &ColumnarRecords) -> bool
self
and other
values to be equal, and is used
by ==
.impl StructuralPartialEq for ColumnarRecords
Auto Trait Implementations§
impl !Freeze for ColumnarRecords
impl RefUnwindSafe for ColumnarRecords
impl Send for ColumnarRecords
impl Sync for ColumnarRecords
impl Unpin for ColumnarRecords
impl UnwindSafe for ColumnarRecords
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<R, O, T> CopyOnto<ConsecutiveOffsetPairs<R, O>> for T
impl<R, O, T> CopyOnto<ConsecutiveOffsetPairs<R, O>> for T
source§fn copy_onto(
self,
target: &mut ConsecutiveOffsetPairs<R, O>
) -> <ConsecutiveOffsetPairs<R, O> as Region>::Index
fn copy_onto( self, target: &mut ConsecutiveOffsetPairs<R, O> ) -> <ConsecutiveOffsetPairs<R, O> as Region>::Index
source§impl<T> FutureExt for T
impl<T> FutureExt for T
source§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
source§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
source§impl<T> Instrument for T
impl<T> Instrument for T
source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
source§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T
in a tonic::Request
source§impl<T> Pointable for T
impl<T> Pointable for T
source§impl<T> ProgressEventTimestamp for T
impl<T> ProgressEventTimestamp for T
source§impl<P, R> ProtoType<R> for Pwhere
R: RustType<P>,
impl<P, R> ProtoType<R> for Pwhere
R: RustType<P>,
source§fn into_rust(self) -> Result<R, TryFromProtoError>
fn into_rust(self) -> Result<R, TryFromProtoError>
RustType::from_proto
.source§fn from_rust(rust: &R) -> P
fn from_rust(rust: &R) -> P
RustType::into_proto
.