Struct dataflow::render::debezium::DebeziumDeduplicationState

struct DebeziumDeduplicationState {
    last_position_and_offset: Option<(RowCoordinates, Option<i64>)>,
    full: Option<TrackFull>,
    messages_processed: u64,
    filenames_to_indices: HashMap<String, i64>,
    projection: DebeziumDedupProjection,
}

Track whether or not we should skip a specific debezium message

The goal of deduplication is to omit sending true duplicates – the exact same record being sent into materialize twice. That means that we create one deduplicator per timely worker and use use timely key sharding normally. But it also means that no single deduplicator knows the highest-ever seen binlog offset.

Fields

last_position_and_offset: Option<(RowCoordinates, Option<i64>)>

Last recorded binlog position and connector offset

DebeziumEnvelope determines whether messages that are not ahead of the last recorded position will be skipped.

full: Option<TrackFull>

Whether or not to track every message we’ve ever seen

messages_processed: u64

filenames_to_indices: HashMap<String, i64>

projection: DebeziumDedupProjection

Implementations

impl DebeziumDeduplicationState

fn new(envelope: DebeziumEnvelope) -> Option<Self>

fn extract_total_order(&mut self, value: &Row) -> Option<i64>

fn extract_binlog_position(
    &mut self,
    value: &Row
) -> Result<Option<RowCoordinates>, DataflowError>

fn should_use_record(
    &mut self,
    key: Option<Row>,
    value: &Row,
    connector_offset: Option<i64>,
    upstream_time_millis: Option<i64>,
    debug_name: &str
) -> Result<bool, DataflowError>

Trait Implementations

impl Debug for DebeziumDeduplicationState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.