Trait dataflow::source::SourceReader[−][src]

pub(crate) trait SourceReader {
    type Key: Data + MaybeLength;
    type Value: Data + MaybeLength;
    fn new(
        source_name: String, 
        source_id: SourceInstanceId, 
        worker_id: usize, 
        worker_count: usize, 
        consumer_activator: SyncActivator, 
        connector: ExternalSourceConnector, 
        restored_offsets: Vec<(PartitionId, Option<MzOffset>)>, 
        encoding: SourceDataEncoding, 
        logger: Option<Logger>, 
        metrics: SourceBaseMetrics
    ) -> Result<(Self, Option<PartitionId>), Error>
    where
        Self: Sized;

    fn get_next_message(
        &mut self
    ) -> Result<NextMessage<Self::Key, Self::Value>, Error>;
}
Expand description

This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.

TODO: this trait is still a little too Kafka-centric, specifically the concept of a “partition” is baked into this trait and introduces some cognitive overhead as we are forced to treat things like file sources as “single-partition”

Associated Types

Required methods

Create a new source reader.

This function returns the source reader and optionally, any “partition” it’s already reading. In practice, the partition is only non-None for static sources that either don’t truly have partitions or have a fixed number of partitions.

Returns the next message available from the source.

Note that implementers are required to present messages in strictly ascending
offset order within each partition.

Implementors