Struct dataflow::source::timestamp::TimestampBindingBox

pub struct TimestampBindingBox {
    known_partitions: HashMap<PartitionId, Option<MzOffset>>,
    partitions: HashMap<PartitionId, PartitionTimestamps>,
    compaction_frontier: MutableAntichain<Timestamp>,
    durability_frontier: Antichain<Timestamp>,
    proposer: TimestampProposer,
    pub activators: Vec<Activator>,
}

This struct holds per-source timestamp state in a way that can be shared across different source instances and allow different source instances to indicate how far they have read up to.

This type is almost never meant to be used directly, and you probably want to use TimestampBindingRc instead.

Fields

known_partitions: HashMap<PartitionId, Option<MzOffset>>

List of partitions that we learned about from the coordinator. This is used by source operators to learn about new partition assignments, it is purely a conduit for getting information from the coordinator to individual source operators.

Note: This is a bit of a hack, in the same way that we used partitions() before to forward new partitions from coordinator to source operator. We could factor this out of TimestampBinding* into it’s own piece that only deals with managing new partitions.

partitions: HashMap<PartitionId, PartitionTimestamps>

List of timestamp bindings per independent partition. This vector is sorted by timestamp and offset and each (time, offset) entry indicates that offsets <= offset should be assigned time as their timestamp. Consecutive entries form an interval of offsets.

compaction_frontier: MutableAntichain<Timestamp>

Indicates the lowest timestamp across all partitions that we retain bindings for. This frontier can be held back by other entities holding the shared TimestampBindingRc.

durability_frontier: Antichain<Timestamp>

Indicates the lowest timestamp across all partititions and across all workers that has been durably persisted.

proposer: TimestampProposer

Generates new timestamps for RT sources

activators: Vec<Activator>

Source operators that should be activated on durability changes.

Implementations

impl TimestampBindingBox

fn new(timestamp_update_interval: u64, now: NowFn) -> Self

fn adjust_compaction_frontier(
    &mut self,
    remove: AntichainRef<'_, Timestamp>,
    add: AntichainRef<'_, Timestamp>
)

fn set_durability_frontier(&mut self, new_frontier: AntichainRef<'_, Timestamp>)

fn compact(&mut self)

fn add_partition(
    &mut self,
    partition: PartitionId,
    restored_offset: Option<MzOffset>
)

fn add_binding(
    &mut self,
    partition: PartitionId,
    timestamp: Timestamp,
    offset: MzOffset
)

fn downgrade(
    &self,
    cap: &mut Capability<Timestamp>,
    cursors: &HashMap<PartitionId, MzOffset>
)

fn get_or_propose_binding(
    &mut self,
    partition: &PartitionId,
    offset: MzOffset
) -> Timestamp

fn get_bindings_in_range(
    &self,
    lower: AntichainRef<'_, Timestamp>,
    upper: AntichainRef<'_, Timestamp>
) -> Vec<(PartitionId, Timestamp, MzOffset)>

fn upper(&self) -> Timestamp

fn read_upper(&self, target: &mut Antichain<Timestamp>)

fn partitions(&self) -> Vec<(PartitionId, Option<MzOffset>)>

fn update_timestamp(&mut self)

Trait Implementations

impl Debug for TimestampBindingBox

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.