Trait mz_storage::source::types::SourceReader

source ·

pub trait SourceReader {
    type Key: Data + MaybeLength;
    type Value: Data + MaybeLength;
    type Time: SourceTimestamp;
    type Diff: Data + Semigroup;

    fn next<'life0, 'async_trait>(
        &'life0 mut self,
        timestamp_granularity: Duration
    ) -> Pin<Box<dyn Future<Output = Option<SourceMessageType<Self::Key, Self::Value, Self::Time, Self::Diff>>> + 'async_trait>>
    where
        Self: 'async_trait,
        'life0: 'async_trait,
    { ... }
    fn get_next_message(
        &mut self
    ) -> NextMessage<Self::Key, Self::Value, Self::Time, Self::Diff> { ... }
    fn into_stream<'a>(
        self,
        timestamp_granularity: Duration
    ) -> LocalBoxStream<'a, SourceMessageType<Self::Key, Self::Value, Self::Time, Self::Diff>>
    where
        Self: Sized + 'a,
    { ... }
}

Expand description

This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.

Contract between `SourceReader` and the ingestion framework

A source reader uses updates emitted from SourceReader::next/SourceReader::get_next_message to update the ingestion framework about new updates retrieved from the external system and about its internal state.

The framework will spawn a SourceReader on each timely worker. It is the responsibility of the reader to figure out which of the partitions (if any) it is responsible for reading using crate::source::responsible_for.