Struct dataflow_types::types::sources::persistence::SourcePersistDesc

pub struct SourcePersistDesc {
    pub upper_seal_ts: u64,
    pub since_ts: u64,
    pub primary_stream: String,
    pub timestamp_bindings_stream: String,
    pub envelope_desc: EnvelopePersistDesc,
}

The details needed to make a source that uses an external super::SourceConnector persistent.

Fields

upper_seal_ts: u64

The current upper seal timestamp of all involved streams.

NOTE: This timestamp is determined when the coordinator starts up or when the source is initially created. When a source is actively writing to this stream, the seal timestamp will progress beyond this timestamp.

This is okay for now because we only want to allow one source instantiation for persistent sources, meaning the flow is usually this:

1. coordinator determines seal timestamp
2. seal timestamps for a source are sent to dataflow when rendering a source
3. coordinator (or anyone) never looks at this timestamp again.

And when we restart, we start from step 1., at which time we are guaranteed not to have a source running already.

since_ts: u64

The current compaction frontier (aka since) of all involved streams.

NOTE: This timestamp is determined when the coordinator starts up or when the source is initially created. When a source is actively writing to this stream and allowing compaction, this will progress beyond this timestamp.

This is okay for now because we only want to allow one source instantiation for persistent sources, meaning the flow is usually this:

1. coordinator determines since timestamp
2. timestamps for a source are sent to dataflow when rendering a source
3. coordinator (or anyone) never looks at this timestamp again.

And when we restart, we start from step 1., at which time we are guaranteed not to have a source running already.

primary_stream: String

Name of the primary persisted stream of this source. This is what a consumer of the persisted data would be interested in while the secondary stream(s) of the source are an internal implementation detail.

timestamp_bindings_stream: String

Persisted stream of timestamp bindings.

envelope_desc: EnvelopePersistDesc

Any additional details that we need to make the envelope logic stateful.

Trait Implementations

impl Clone for SourcePersistDesc

fn clone(&self) -> SourcePersistDesc

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SourcePersistDesc

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for SourcePersistDesc

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl PartialEq<SourcePersistDesc> for SourcePersistDesc

fn eq(&self, other: &SourcePersistDesc) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &SourcePersistDesc) -> bool

This method tests for !=.

impl Serialize for SourcePersistDesc

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl Eq for SourcePersistDesc

impl StructuralEq for SourcePersistDesc

impl StructuralPartialEq for SourcePersistDesc