Trait mz_storage_operators::s3_oneshot_sink::CopyToS3Uploader

trait CopyToS3Uploader: Sized {
    // Required methods
    fn new(
        sdk_config: SdkConfig,
        connection_details: S3UploadInfo,
        sink_id: &GlobalId,
        batch: u64,
        params: CopyToParameters,
    ) -> Result<Self, Error>;
    async fn append_row(&mut self, row: &Row) -> Result<(), Error>;
    async fn finish(&mut self) -> Result<(), Error>;
}

This trait is used to abstract over the upload details for different file formats. Each format has its own buffering semantics and upload logic, since some can be written in a streaming fashion row-by-row, whereas others use a columnar-based format that requires buffering a batch of rows before writing to S3.

Required Methods

fn new( sdk_config: SdkConfig, connection_details: S3UploadInfo, sink_id: &GlobalId, batch: u64, params: CopyToParameters, ) -> Result<Self, Error>

async fn append_row(&mut self, row: &Row) -> Result<(), Error>

Append a row to the internal buffer, and optionally flush the buffer to S3.

async fn finish(&mut self) -> Result<(), Error>

Flush the full remaining internal buffer to S3, and close all open resources. This will be called when the input stream is finished.

Object Safety

This trait is not object safe.

Implementors

impl CopyToS3Uploader for ParquetUploader

impl CopyToS3Uploader for PgCopyUploader