Struct dataflow::source::SourceOutput

pub struct SourceOutput<K, V> where
    K: Data,
    V: Data,  {
    pub key: K,
    pub value: V,
    pub position: Option<i64>,
    pub upstream_time_millis: Option<i64>,
    pub partition: PartitionId,
}

A record produced by a source

Fields

key: K

The record’s key (or some empty/default value for sources without the concept of key)

value: V

The record’s value

position: Option<i64>

The position in the source, if such a concept exists (e.g., Kafka offset, file line number)

upstream_time_millis: Option<i64>

The time the record was created in the upstream systsem, as milliseconds since the epoch

partition: PartitionId

The partition of this message, present iff the partition comes from Kafka

Implementations

impl<K, V> SourceOutput<K, V> where
    K: Data,
    V: Data, 

pub fn new(
    key: K,
    value: V,
    position: Option<i64>,
    upstream_time_millis: Option<i64>,
    partition: PartitionId
) -> SourceOutput<K, V>

Build a new SourceOutput

impl<K, V> SourceOutput<K, V> where
    K: Data + Serialize + for<'a> Deserialize<'a> + Send + Sync,
    V: Data + Serialize + for<'a> Deserialize<'a> + Send + Sync, 

pub fn key_contract() -> impl ParallelizationContract<Timestamp, Self> where
    K: Hashable<Output = u64>, 

A parallelization contract that hashes by keys.

It would not be ideal to use this in a case where the keys might be empty (or for any other reason too often the same), as that would lead to serial execution of whatever operator is being “parallelized” by this contract.

One good reason to use this (and as of this writing, apparently the only reason) is for operators that need to do some dedpulication logic based on keys: For example, upserts, or Debezium dedpulication based on MySQL binlog coordinates.

pub fn position_value_contract(
) -> impl ParallelizationContract<Timestamp, Self> where
    V: Hashable<Output = u64>, 

A parallelization contract that hashes by positions (if available) and otherwise falls back to hashing by value. Values can be just as skewed as keys, whereas positions are generally known to be unique or close to unique in a source. For example, Kafka offsets are unique per-partition. Most decode logic should use this instead of key_contract.

Trait Implementations

impl<K: Clone, V: Clone> Clone for SourceOutput<K, V> where
    K: Data,
    V: Data, 

fn clone(&self) -> SourceOutput<K, V>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<K: Debug, V: Debug> Debug for SourceOutput<K, V> where
    K: Data,
    V: Data, 

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de, K, V> Deserialize<'de> for SourceOutput<K, V> where
    K: Data,
    V: Data,
    K: Deserialize<'de>,
    V: Deserialize<'de>, 

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl<K, V> Serialize for SourceOutput<K, V> where
    K: Data,
    V: Data,
    K: Serialize,
    V: Serialize, 

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.