Trait CopyToS3Uploader

trait CopyToS3Uploader: Sized {
    // Required methods
    fn new(
        sdk_config: SdkConfig,
        connection_details: S3UploadInfo,
        sink_id: &GlobalId,
        batch: u64,
        params: CopyToParameters,
    ) -> Result<Self, Error>;
    async fn force_new_file(&mut self) -> Result<(), Error>;
    async fn append_row(&mut self, row: &Row) -> Result<(), Error>;
    async fn finish(&mut self) -> Result<(), Error>;
}

This trait is used to abstract over the upload details for different file formats. Each format has its own buffering semantics and upload logic, since some can be written in a streaming fashion row-by-row, whereas others use a columnar-based format that requires buffering a batch of rows before writing to S3.

Required Methods

fn new( sdk_config: SdkConfig, connection_details: S3UploadInfo, sink_id: &GlobalId, batch: u64, params: CopyToParameters, ) -> Result<Self, Error>

async fn force_new_file(&mut self) -> Result<(), Error>

Force the start of a new file, even if no rows have yet been appended or if the current file has not yet reached the configured max_file_size.

async fn append_row(&mut self, row: &Row) -> Result<(), Error>

Append a row to the internal buffer, and optionally flush the buffer to S3.

async fn finish(&mut self) -> Result<(), Error>

Flush the full remaining internal buffer to S3, and close all open resources. This will be called when the input stream is finished.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl CopyToS3Uploader for ParquetUploader

impl CopyToS3Uploader for PgCopyUploader