Struct mz_aws_util::s3_uploader::S3MultiPartUploader

pub struct S3MultiPartUploader {
    client: Client,
    config: S3MultiPartUploaderConfig,
    bucket: String,
    key: String,
    upload_id: String,
    part_count: i32,
    total_bytes_uploaded: u64,
    buffer: BytesMut,
    upload_handles: Vec<JoinHandle<Result<(Option<String>, i32), S3MultiPartUploadError>>>,
}

A multi part uploader which can upload a single object across multiple parts and keeps track of state to eventually finish the upload process. The caller does not need to know the final number of parts beforehand.

The caller should get an instance by calling S3MultiPartUploader::try_new first. Each part can be added by calling add_chunk, and can be called one or more times and eventually finish the multi part upload by calling finish method.

Fields

client: Client

config: S3MultiPartUploaderConfig

bucket: String

key: String

upload_id: String

part_count: i32

total_bytes_uploaded: u64

buffer: BytesMut

upload_handles: Vec<JoinHandle<Result<(Option<String>, i32), S3MultiPartUploadError>>>

Implementations

impl S3MultiPartUploader

pub async fn try_new( sdk_config: &SdkConfig, bucket: String, key: String, config: S3MultiPartUploaderConfig, ) -> Result<S3MultiPartUploader, S3MultiPartUploadError>

Creates a an instance of S3MultiPartUploader for the given bucket and path. This starts the multi part upload by making a create_multipart_upload call, and initializes all the internal state required to track the ongoing upload.

pub fn buffer_chunk( &mut self, data: &[u8], ) -> Result<(), S3MultiPartUploadError>

Adds the data to the internal buffer and flushes the buffer if it is more than the part threshold defined in S3MultiPartUploaderConfig. Returns an UploadExceedsMaxFileLimit error if the upload will exceed the configured file_size_limit, unless no data has been added yet. In which case, it will try to do an upload if the data size is under part_size_limit * 10000.

pub async fn finish(self) -> Result<CompletedUpload, S3MultiPartUploadError>

Method to finish the multi part upload. If the buffer is not empty, it flushes the buffer first and then makes a call to complete_multipart_upload. Returns the number of parts and number of bytes uploaded.

fn buffer_size(&self) -> u64

fn remaining_bytes_limit(&self) -> u64

Internal method, returns the amount of bytes which can still be added to the multi-part upload. without exceeding file_size_limit.

pub fn added_bytes(&self) -> u64

Internal method, returns the number of bytes processed till now.

fn flush_chunks(&mut self) -> Result<(), S3MultiPartUploadError>

Internal method to continuously flush and upload part from the buffer till it is under the configured part_size_limit.

fn upload_part_internal( &mut self, data: Bytes, ) -> Result<(), S3MultiPartUploadError>

Internal method which actually uploads a single part and updates state.

Trait Implementations

impl Debug for S3MultiPartUploader

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.