Enum FormatKind

pub(crate) enum FormatKind {
    Csv(CsvDecoder),
    Parquet(ParquetFormat),
}

An enum wrapper around OneshotFormats.

An alternative to this wrapper would be to use Box<dyn OneshotFormat>, but that requires making the trait object safe and it’s easier to just wrap it in an enum. Also, this wrapper provides a convenient place to add StorageErrorXContext::context for all of our format types.

Variants

Csv(CsvDecoder)

Parquet(ParquetFormat)

Trait Implementations

impl Clone for FormatKind

fn clone(&self) -> FormatKind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FormatKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl OneshotFormat for FormatKind

type WorkRequest<S> = RequestKind<<S as OneshotSource>::Object, <S as OneshotSource>::Checksum> where S: OneshotSource

A single unit of work for decoding this format, e.g. a single Parquet RowGroup.

type RecordChunk = RecordChunkKind

A chunk of records in this format that can be decoded into Rows.

async fn split_work<S: OneshotSource + Send>( &self, source: S, object: S::Object, checksum: S::Checksum, ) -> Result<Vec<Self::WorkRequest<S>>, StorageErrorX>

Given an upstream object, defines how we should parse this object in parallel.

fn fetch_work<'a, S: OneshotSource + Sync + 'static>( &'a self, source: &'a S, request: Self::WorkRequest<S>, ) -> BoxStream<'a, Result<Self::RecordChunk, StorageErrorX>>

Given a work request, fetch data from the OneshotSource and return it in a format that can later be decoded.

fn decode_chunk( &self, chunk: Self::RecordChunk, rows: &mut Vec<Row>, ) -> Result<usize, StorageErrorX>

Decode a chunk of records into Rows.