Struct mz_persist::s3::S3BlobConfig

pub struct S3BlobConfig {
    metrics: S3BlobMetrics,
    client: Client,
    bucket: String,
    prefix: String,
    cfg: Arc<ConfigSet>,
    is_cc_active: bool,
}

Configuration for opening an S3Blob.

Fields

metrics: S3BlobMetrics

client: Client

bucket: String

prefix: String

cfg: Arc<ConfigSet>

is_cc_active: bool

Implementations

impl S3BlobConfig

const EXTERNAL_TESTS_S3_BUCKET: &'static str = "MZ_PERSIST_EXTERNAL_STORAGE_TEST_S3_BUCKET"

pub async fn new( bucket: String, prefix: String, role_arn: Option<String>, endpoint: Option<String>, region: Option<String>, credentials: Option<(String, String)>, knobs: Box<dyn BlobKnobs>, metrics: S3BlobMetrics, cfg: Arc<ConfigSet>, ) -> Result<Self, Error>

Returns a new S3BlobConfig for use in production.

Stores objects in the given bucket prepended with the (possibly empty) prefix. S3 credentials and region must be available in the process or environment.

pub async fn new_for_test() -> Result<Option<Self>, Error>

Returns a new S3BlobConfig for use in unit tests.

By default, persist tests that use external storage (like s3) are no-ops, so that cargo test does the right thing without any configuration. To activate the tests, set the MZ_PERSIST_EXTERNAL_STORAGE_TEST_S3_BUCKET environment variable and ensure you have valid AWS credentials available in a location where the AWS Rust SDK can discovery them.

This intentionally uses the MZ_PERSIST_EXTERNAL_STORAGE_TEST_S3_BUCKET env as the switch for test no-op-ness instead of the presence of a valid AWS authentication configuration envs because a developers might have valid credentials present and this isn’t an explicit enough signal from a developer running cargo test that it’s okay to use these credentials. It also intentionally does not use the local drop-in s3 replacement to keep persist unit tests light.

On CI, these tests are enabled by adding the scratch-aws-access plugin to the cargo-test step in ci/test/pipeline.template.yml and setting MZ_PERSIST_EXTERNAL_STORAGE_TEST_S3_BUCKET in ci/test/cargo-test/mzcompose.py.

For a Materialize developer, to opt in to these tests locally for development, follow the AWS access guide:

https://github.com/MaterializeInc/i2/blob/main/doc/aws-access.md

then running source src/persist/s3_test_env_mz.sh. You will also have to run aws sso login if you haven’t recently.

Non-Materialize developers will have to set up their own auto-deleting bucket and export the same env vars that s3_test_env_mz.sh does.

Only public for use in src/benches.

pub fn clone_with_new_uuid_prefix(&self) -> Self

Returns a clone of Self with a new v4 uuid prefix.

Trait Implementations

impl Clone for S3BlobConfig

fn clone(&self) -> S3BlobConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for S3BlobConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.