Struct mz_persist_client::internal::trace::FlatTrace

pub struct FlatTrace<T> {
    pub(crate) since: Antichain<T>,
    pub(crate) legacy_batches: BTreeMap<Arc<HollowBatch<T>>, ()>,
    pub(crate) hollow_batches: BTreeMap<SpineId, Arc<HollowBatch<T>>>,
    pub(crate) spine_batches: BTreeMap<SpineId, ThinSpineBatch<T>>,
    pub(crate) merges: BTreeMap<SpineId, ThinMerge<T>>,
}

This is a “flattened” representation of a Trace. Goals:

• small updates to the trace should result in small differences in the FlatTrace;
• two FlatTraces should be efficient to diff;
• converting to and from a Trace should be relatively straightforward.

These goals are all somewhat in tension, and the space of possible representations is pretty large. See individual fields for comments on some of the tradeoffs.

Fields

since: Antichain<T>

legacy_batches: BTreeMap<Arc<HollowBatch<T>>, ()>

Hollow batches without an associated ID. If this flattened trace contains spine batches, we can figure out which legacy batch belongs in which spine batch by comparing the descs. Previously, we serialized a trace as just this list of batches. Keeping this data around helps ensure backwards compatibility. In the near future, we may still keep some batches here to help minimize the size of diffs – rewriting all the hollow batches in a shard can be prohibitively expensive. Eventually, we’d like to remove this in favour of the collection below.

hollow_batches: BTreeMap<SpineId, Arc<HollowBatch<T>>>

Hollow batches with an associated ID. Spine batches can reference these hollow batches by id directly.

spine_batches: BTreeMap<SpineId, ThinSpineBatch<T>>

Spine batches stored by ID. We reference hollow batches by ID, instead of inlining them, to make differential updates smaller when two batches merge together. We also store the level on the batch, instead of mapping from level to a list of batches… the level of a spine batch doesn’t change over time, but the list of batches at a particular level does.

merges: BTreeMap<SpineId, ThinMerge<T>>

In-progress merges. We store this by spine id instead of level to prepare for some possible generalizations to spine (merging N of M batches at a level). This is also a natural place to store incremental merge progress in the future.

Trait Implementations

impl<T: Clone> Clone for FlatTrace<T>

fn clone(&self) -> FlatTrace<T>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug> Debug for FlatTrace<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Timestamp + Codec64> RustType<ProtoTrace> for FlatTrace<T>

fn into_proto(&self) -> ProtoTrace

Convert a Self into a Proto value.

fn from_proto(proto: ProtoTrace) -> Result<Self, TryFromProtoError>

Consume and convert a Proto back into a Self value.

fn into_proto_owned(self) -> Proto

A zero clone version of Self::into_proto that types can optionally implement, otherwise, the default implementation delegates to Self::into_proto.