Struct FanoutWriter

pub struct FanoutWriter<B, I = RecordBatch, O = Vec<DataFile>>
where
    B: IcebergWriterBuilder<I, O>,
    O: IntoIterator + FromIterator<<O as IntoIterator>::Item>,
    <O as IntoIterator>::Item: Clone,
{ /* private fields */ }

A writer that can write data to multiple partitions simultaneously.

Unlike ClusteredWriter which expects sorted input and maintains only one active writer, FanoutWriter can handle unsorted data by maintaining multiple active writers in a map. This allows writing to any partition at any time, but uses more memory as all writers remain active until the writer is closed.

Type Parameters

• B - The inner writer builder type
• I - Input type (defaults to RecordBatch)
• O - Output collection type (defaults to Vec<DataFile>)

Implementations

impl<B, I, O> FanoutWriter<B, I, O>

where B: IcebergWriterBuilder<I, O>, I: Send + 'static, O: IntoIterator + FromIterator<<O as IntoIterator>::Item>, <O as IntoIterator>::Item: Send + Clone,

pub fn new(inner_builder: B) -> Self

Create a new FanoutWriter.

Trait Implementations

impl<B, I, O> PartitioningWriter<I, O> for FanoutWriter<B, I, O>

where B: IcebergWriterBuilder<I, O>, I: Send + 'static, O: IntoIterator + FromIterator<<O as IntoIterator>::Item> + Send + 'static, <O as IntoIterator>::Item: Send + Clone,

fn write<'life0, 'async_trait>( &'life0 mut self, partition_key: PartitionKey, input: I, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Write data with a partition key.

fn close<'async_trait>( self, ) -> Pin<Box<dyn Future<Output = Result<O>> + Send + 'async_trait>>

where Self: 'async_trait,

Close the writer and return the output.