Skip to main content

mint_batch_descriptions

Function mint_batch_descriptions 

Source
fn mint_batch_descriptions<'scope>(
    name: String,
    sink_id: GlobalId,
    input: StreamVec<'scope, Timestamp, <TraceAgent<OrdValSpine<Option<Row>, Row, Timestamp, Diff>> as TraceReader>::Batch>,
    sink: &StorageSinkDesc<CollectionMetadata, Timestamp>,
    connection: IcebergSinkConnection,
    storage_configuration: StorageConfiguration,
    initial_schema: SchemaRef,
) -> (StreamVec<'scope, Timestamp, (Antichain<Timestamp>, Antichain<Timestamp>)>, StreamVec<'scope, Timestamp, Infallible>, StreamVec<'scope, Timestamp, HealthStatusMessage>, PressOnDropButton)
Expand description

Generate time-based batch boundaries for grouping writes into Iceberg snapshots. Batches are minted with configurable windows to balance write efficiency with latency. We maintain a sliding window of future batch descriptions so writers can start processing data even while earlier batches are still being written.