Struct mz_storage_operators::s3_oneshot_sink::pgcopy::PgCopyUploader

pub(super) struct PgCopyUploader {
    desc: RelationDesc,
    format: CopyFormatParams<'static>,
    file_index: usize,
    key_manager: S3KeyManager,
    batch: u64,
    max_file_size: u64,
    sdk_config: Option<SdkConfig>,
    current_file_uploader: Option<S3MultiPartUploader>,
    params: CopyToParameters,
}

Required state to upload batches to S3

Fields

desc: RelationDesc

The output description.

format: CopyFormatParams<'static>

Params to format the data.

file_index: usize

The index of the current file within the batch.

key_manager: S3KeyManager

Provides the appropriate bucket and object keys to use for uploads

batch: u64

Identifies the batch that files uploaded by this uploader belong to

max_file_size: u64

The desired file size. A new file upload will be started when the size exceeds this amount.

sdk_config: Option<SdkConfig>

The aws sdk config. This is an option so that we can get an owned value later to move to a spawned tokio task.

current_file_uploader: Option<S3MultiPartUploader>

Multi-part uploader for the current file. Keeping the uploader in an Option to later take owned value.

params: CopyToParameters

Upload parameters.

Implementations

impl PgCopyUploader

async fn start_new_file_upload(&mut self) -> Result<(), Error>

Creates the uploader for the next file and starts the multi part upload.

Trait Implementations

impl CopyToS3Uploader for PgCopyUploader

async fn finish(&mut self) -> Result<(), Error>

Finishes any remaining in-progress upload.

async fn append_row(&mut self, row: &Row) -> Result<(), Error>

Appends the row to the in-progress upload where it is buffered till it reaches the configured part_size_limit after which the S3MultiPartUploader will upload that part. In case it will exceed the max file size of the ongoing upload, then a new S3MultiPartUploader for a new file will be created and the row data will be appended there.

fn new( sdk_config: SdkConfig, connection_details: S3UploadInfo, sink_id: &GlobalId, batch: u64, params: CopyToParameters, ) -> Result<PgCopyUploader, Error>