Trait mz_storage::source::SourceReader

source · [−]
pub trait SourceReader {
    type Key: Data + MaybeLength;
    type Value: Data + MaybeLength;
    type Diff: Data;

    fn new(
        source_name: String, 
        source_id: GlobalId, 
        worker_id: usize, 
        worker_count: usize, 
        consumer_activator: SyncActivator, 
        connection: SourceConnection, 
        restored_offsets: Vec<(PartitionId, Option<MzOffset>)>, 
        encoding: SourceDataEncoding, 
        metrics: SourceBaseMetrics, 
        connection_context: ConnectionContext
    ) -> Result<Self, Error>
    where
        Self: Sized;

    fn next<'life0, 'async_trait>(
        &'life0 mut self, 
        timestamp_interval: Duration
    ) -> Pin<Box<dyn Future<Output = Option<Result<SourceMessageType<Self::Key, Self::Value, Self::Diff>, SourceReaderError>>> + 'async_trait>>
    where
        'life0: 'async_trait,
        Self: 'async_trait,
    { ... }
    fn get_next_message(
        &mut self
    ) -> Result<NextMessage<Self::Key, Self::Value, Self::Diff>, SourceReaderError> { ... }
    fn into_stream<'a>(
        self, 
        timestamp_interval: Duration
    ) -> LocalBoxStream<'a, Result<SourceMessageType<Self::Key, Self::Value, Self::Diff>, SourceReaderError>>
    where
        Self: Sized + 'a,
    { ... }
}
Expand description

This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.

TODO: this trait is still a little too Kafka-centric, specifically the concept of a “partition” is baked into this trait and introduces some cognitive overhead as we are forced to treat things like file sources as “single-partition”

Required Associated Types

Required Methods

Create a new source reader.

This function returns the source reader and optionally, any “partition” it’s already reading. In practice, the partition is only non-None for static sources that either don’t truly have partitions or have a fixed number of partitions.

Provided Methods

Returns the next message available from the source.

Note that implementers are required to present messages in strictly ascending offset order within each partition.

Returns the next message available from the source.

Note that implementers are required to present messages in strictly ascending offset order within each partition.

Deprecated

Source implementation should implement the async SourceReader::next method instead.

Returns an adapter that treats the source as a stream.

The stream produces the messages that would be produced by repeated calls to next.

Implementors