Skip to main content

OneshotFormat

mz_storage_operators::oneshot_source

Trait OneshotFormat 

Source 
pub trait OneshotFormat: Clone {
    type WorkRequest<S>: Debug + Clone + Send + Serialize + DeserializeOwned + 'static
       where S: OneshotSource;
    type RecordChunk: Debug + Clone + Send + Serialize + DeserializeOwned + 'static;

    // Required methods
    fn split_work<S: OneshotSource + Send>(
        &self,
        source: S,
        object: S::Object,
        checksum: S::Checksum,
    ) -> impl Future<Output = Result<Vec<Self::WorkRequest<S>>, StorageErrorX>> + Send;
    fn fetch_work<'a, S: OneshotSource + Sync + 'static>(
        &'a self,
        source: &'a S,
        request: Self::WorkRequest<S>,
    ) -> BoxStream<'a, Result<Self::RecordChunk, StorageErrorX>>;
    fn decode_chunk(
        &self,
        chunk: Self::RecordChunk,
        rows: &mut Vec<Row>,
    ) -> Result<usize, StorageErrorX>;
}
Expand description

Defines a format that we fetch for a “one time” ingestion.

Required Associated Types§

Source

type WorkRequest<S>: Debug + Clone + Send + Serialize + DeserializeOwned + 'static where S: OneshotSource

A single unit of work for decoding this format, e.g. a single Parquet RowGroup.

Source

type RecordChunk: Debug + Clone + Send + Serialize + DeserializeOwned + 'static

A chunk of records in this format that can be decoded into Rows.

Required Methods§

Source

fn split_work<S: OneshotSource + Send>( &self, source: S, object: S::Object, checksum: S::Checksum, ) -> impl Future<Output = Result<Vec<Self::WorkRequest<S>>, StorageErrorX>> + Send

Given an upstream object, defines how we should parse this object in parallel.

Note: It’s totally fine to not process an object in parallel, and just return a single Self::WorkRequest here.

Source

fn fetch_work<'a, S: OneshotSource + Sync + 'static>( &'a self, source: &'a S, request: Self::WorkRequest<S>, ) -> BoxStream<'a, Result<Self::RecordChunk, StorageErrorX>>

Given a work request, fetch data from the OneshotSource and return it in a format that can later be decoded.

Source

fn decode_chunk( &self, chunk: Self::RecordChunk, rows: &mut Vec<Row>, ) -> Result<usize, StorageErrorX>

Decode a chunk of records into Rows.

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§