Struct mz_storage_client::util::antichain::OffsetAntichain

pub struct OffsetAntichain {
    inner: BTreeMap<PartitionId, MzOffset>,
}

OffsetAntichain is similar to a timely Antichain<(PartitionId, T: TotalOrder)>, but additionally:

• Uses a BTreeMap as the implementation to allow absence of a PartitionId to mean that PartitionId is at T::minimum. This helps avoid needing to hold onto a HUGE Antichain for all possible PartitionIds
  • Note this means that a partition being “finished” (like a normal “empty” Antichain, is not currently supported, but could be added
  • Note that this Antichain can also have been filtered, as in, missing some partitions for which data exists but we don’t care about. This is semantically different than if we just don’t have data, but it is represented the same
• Is not generic over T, but instead uses MzOffset, which:
  • implements TotalOrder
  • implements checked_sub
• Allows users to go from a frontier to an actual set of offsets that are connected to real data.
  • This is a consequence of implementation, where the frontier is ALWAYS generated from real data offsets, in an invertible way.

OffsetAntichain has 4 sets of Api’s:

• “read” apis like get and as_vec
• “mutation” apis (currently only filter_by_partition)
• And 2 “write” apis, that should primarily be used separately from each other:
  • “Frontier” apis, which directly manipulate the underlying frontier. Useful for implementing primitives like reclocking
  • “Data” apis, that maintain special invariants:
    • insert_data_up_to updates the frontier based on a given offset that is associated with actual data.
    • as_data_offsets inverts the behavior of insert_data_up_to and returns a BTreeMap<PartitionId, MzOffset> of offets of real committed data.

Fields

inner: BTreeMap<PartitionId, MzOffset>

Implementations

impl OffsetAntichain

pub fn new() -> Self

Initialize an Antichain where all partitions have made no progress.

pub fn as_data_offsets(&self) -> BTreeMap<PartitionId, MzOffset>

Produce offsets for all partitions in this OffsetAntichain that were at one point given by insert_data_up_to.

If the partition is yet to make any progress, it may be filtered out.

Invariant: After initialization, only insert_data_up_to (not insert and friends) may be used with this OffsetAntichain for this function to produce meaningful values, unless you are very careful.

pub fn insert(&mut self, pid: PartitionId, m: MzOffset) -> Option<MzOffset>

Insert a new MzOffset frontier value for the pid, returning the old one if it wasn’t there.

pub fn maybe_insert(&mut self, pid: PartitionId, offset: MzOffset)

Insert a new MzOffset frontier value for pid if it is larger than the previously stored value.

pub fn extend<T: IntoIterator<Item = (PartitionId, MzOffset)>>(
    &mut self,
    iter: T
)

The same as insert, but for many values.

pub fn advance(&mut self, pid: PartitionId, diff: MzOffset)

Advance the frontier for PartitionId by diff Initializes the offset for pid if it doesn’t exist.

pub fn less_equal(&self, other: &OffsetAntichain) -> bool

Returns true iff this OffsetAntichain is <= other. This requires that for all partitions tracked in other, the offset in self is <= or absent.

pub fn bounded(&self, other: &OffsetAntichain) -> OffsetAntichain

Creates a new OffsetAntichain that starts out as a copy of self but where each offset is upper bounded by the corresponding offset from other, if there is one.

NOTE: This is not an equivalent of meet, as known from timely Antichain. This operation is asymmetric: we want partitions in the result only if they exist in self, we don’t want partitions in the result that only exist in other. If we did the latter, this could mean that we advance a frontier further than the original self` would have.

pub fn get(&self, pid: &PartitionId) -> Option<&MzOffset>

Attempt to the the MzOffset value for pid’s frontier

pub fn partitions(&self) -> impl Iterator<Item = &PartitionId>

List the contained partitions.

pub fn iter(&self) -> impl Iterator<Item = (&PartitionId, &MzOffset)>

Iterate over the entire frontier.

pub fn into_iter(self) -> impl Iterator<Item = (PartitionId, MzOffset)>

Iterate over the entire frontier.

pub fn as_vec(&self) -> Vec<(PartitionId, Option<MzOffset>)>

Convert the frontier into a vector. Useful for certain old apis in the storage crate.

pub fn filter_by_partition<F>(&mut self, filter: F)where
    F: FnMut(&PartitionId) -> bool,

Scope this OffsetAntichain down to only partitions that pass this filter callback.

Trait Implementations

impl Clone for OffsetAntichain

fn clone(&self) -> OffsetAntichain

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OffsetAntichain

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for OffsetAntichain

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
    __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<Antichain<Partitioned<i32, MzOffset>>> for OffsetAntichain

Implementation that converts from an Antichain to an OffsetAntichain. This translation code is here for compatibility and will be removed once the pipeline is switched to native timestamps throughout.

Native antichains can express a superset of frontiers that the OffsetAntichain can, therefore the conversion here can fail. While it’s non-standard to panic in From implementations this is fine here because it’s just an interim step that will be removed shortly and all uses would have to panic on error anyway.

fn from(frontier: Antichain<Partitioned<i32, MzOffset>>) -> Self

Converts to this type from the input type.

impl From<OffsetAntichain> for Antichain<Partitioned<i32, MzOffset>>

Implementation that converts from an OffsetAntichain to an Antichain. This translation code is here for compatibility and will be removed once the pipeline is switched to native timestamps throughout.

fn from(frontier: OffsetAntichain) -> Self

Converts to this type from the input type.

impl PartialEq<BTreeMap<PartitionId, MzOffset, Global>> for OffsetAntichain

fn eq(&self, other: &BTreeMap<PartitionId, MzOffset>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<OffsetAntichain> for OffsetAntichain

fn eq(&self, other: &OffsetAntichain) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for OffsetAntichain

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>where
    __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for OffsetAntichain