Struct DeltaWriter

pub struct DeltaWriter<DW, PDW, EDW> {
    pub data_writer: DW,
    pub pos_delete_writer: PDW,
    pub eq_delete_writer: EDW,
    pub unique_cols: Vec<i32>,
    pub seen_rows: HashMap<OwnedRow, Position>,
    /* private fields */
}

A writer that handles row-level changes by combining data file and delete file writers.

Fields

data_writer: DW

The data file writer for new and updated rows.

pos_delete_writer: PDW

The position delete file writer for deletions of existing rows (that have been written within this writer).

eq_delete_writer: EDW

The equality delete file writer for deletions of rows based on equality conditions (for rows that may exist in other data files).

unique_cols: Vec<i32>

The list of unique columns used for equality deletes.

seen_rows: HashMap<OwnedRow, Position>

A map of rows (projected to unique columns) to their corresponding position information.

Trait Implementations

impl<DW, PDW, EDW> IcebergWriter for DeltaWriter<DW, PDW, EDW>

where DW: IcebergWriter + CurrentFileStatus, PDW: IcebergWriter, EDW: IcebergWriter,

fn write<'life0, 'async_trait>( &'life0 mut self, batch: RecordBatch, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Write data to iceberg table.

fn close<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<Vec<DataFile>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Close the writer and return the written data files. If close failed, the data written before maybe be lost. User may need to recreate the writer and rewrite the data again.