Struct mz_persist_client::cfg::PersistConfig

source ·

pub struct PersistConfig {Show 19 fields
    pub(crate) build_version: Version,
    pub hostname: String,
    pub now: NowFn,
    pub dynamic: Arc<DynamicConfig>,
    pub compaction_enabled: bool,
    pub compaction_concurrency_limit: usize,
    pub compaction_queue_size: usize,
    pub compaction_yield_after_n_updates: usize,
    pub consensus_connection_pool_max_size: usize,
    pub consensus_connection_pool_max_wait: Option<Duration>,
    pub writer_lease_duration: Duration,
    pub critical_downgrade_interval: Duration,
    pub pubsub_connect_attempt_timeout: Duration,
    pub pubsub_connect_max_backoff: Duration,
    pub pubsub_client_sender_channel_size: usize,
    pub pubsub_client_receiver_channel_size: usize,
    pub pubsub_server_connection_channel_size: usize,
    pub pubsub_state_cache_shard_ref_channel_size: usize,
    pub pubsub_reconnect_backoff: Duration,
}

Expand description

The tunable knobs for persist.

Tuning inputs:

A larger blob_target_size (capped at KEY_VAL_DATA_MAX_LEN) results in fewer entries in consensus state. Before we have compaction and/or incremental state, it is already growing without bound, so this is a concern. OTOH, for any “reasonable” size (> 100MiB?) of blob_target_size, it seems we’d end up with a pretty tremendous amount of data in the shard before this became a real issue.
A larger blob_target_size will results in fewer s3 operations, which are charged per operation. (Hmm, maybe not if we’re charged per call in a multipart op. The S3Blob impl already chunks things at 8MiB.)
A smaller blob_target_size will result in more even memory usage in readers.
A larger batch_builder_max_outstanding_parts increases throughput (to a point).
A smaller batch_builder_max_outstanding_parts provides a bound on the amount of memory used by a writer.
A larger compaction_heuristic_min_inputs means state size is larger.
A smaller compaction_heuristic_min_inputs means more compactions happen (higher write amp).
A larger compaction_heuristic_min_updates means more consolidations are discovered while reading a snapshot (higher read amp and higher space amp).
A smaller compaction_heuristic_min_updates means more compactions happen (higher write amp).

Tuning logic:

blob_target_size was initially selected to be an exact multiple of 8MiB (the s3 multipart size) that was in the same neighborhood as our initial max throughput (~250MiB).
batch_builder_max_outstanding_parts was initially selected to be as small as possible without harming pipelining. 0 means no pipelining, 1 is full pipelining as long as generating data takes less time than writing to s3 (hopefully a fair assumption), 2 is a little extra slop on top of 1.
compaction_heuristic_min_inputs was set by running the open-loop benchmark with batches of size 10,240 bytes (selected to be small but such that the overhead of our columnar encoding format was less than 10%) and manually increased until the write amp stopped going down. This becomes much less important once we have incremental state. The initial value is a placeholder and should be revisited at some point.
compaction_heuristic_min_updates was set via a thought experiment. This is an O(n*log(n)) upper bound on the number of unconsolidated updates that would be consolidated if we compacted as the in-mem Spine does. The initial value is a placeholder and should be revisited at some point.

TODO: Move these tuning notes into SessionVar descriptions once we have SystemVars for most of these.

Fields§

§build_version: Version

Info about which version of the code is running.

§hostname: String

Hostname of this persist user. Stored in state and used for debugging.

§now: NowFn

A clock to use for all leasing and other non-debugging use.

§dynamic: Arc<DynamicConfig>

Configurations that can be dynamically updated.

§compaction_enabled: bool

Whether to physically and logically compact batches in blob storage.

§compaction_concurrency_limit: usize

In Compactor::compact_and_apply_background, the maximum number of concurrent compaction requests that can execute for a given shard.

§compaction_queue_size: usize

In Compactor::compact_and_apply_background, the maximum number of pending compaction requests to queue.

§compaction_yield_after_n_updates: usize

In Compactor::compact_and_apply_background, how many updates to encode or decode before voluntarily yielding the task.

§consensus_connection_pool_max_size: usize

The maximum size of the connection pool to Postgres/CRDB when performing consensus reads and writes.

§consensus_connection_pool_max_wait: Option<Duration>

The maximum time to wait when attempting to obtain a connection from the pool.

§writer_lease_duration: Duration

Length of time after a writer’s last operation after which the writer may be expired.

§critical_downgrade_interval: Duration

Length of time between critical handles’ calls to downgrade since

§pubsub_connect_attempt_timeout: Duration

Timeout per connection attempt to Persist PubSub service.

§pubsub_connect_max_backoff: Duration

Maximum backoff when retrying connection establishment to Persist PubSub service.

§pubsub_client_sender_channel_size: usize

Size of channel used to buffer send messages to PubSub service.

§pubsub_client_receiver_channel_size: usize

Size of channel used to buffer received messages from PubSub service.

§pubsub_server_connection_channel_size: usize

Size of channel used per connection to buffer broadcasted messages from PubSub server.

§pubsub_state_cache_shard_ref_channel_size: usize

Size of channel used by the state cache to broadcast shard state references.

§pubsub_reconnect_backoff: Duration

Backoff after an established connection to Persist PubSub service fails.

Implementations§

source §

impl PersistConfig

source

pub fn new(build_info: &BuildInfo, now: NowFn) -> Self

Returns a new instance of PersistConfig with default tuning.

source

pub fn sink_minimum_batch_updates(&self) -> usize

The minimum number of updates that justify writing out a batch in persist_sink’s write_batches operator. (If there are fewer than this minimum number of updates, they’ll be forwarded on to append_batch to be combined and written there.)

source

pub fn storage_sink_minimum_batch_updates(&self) -> usize

The same as Self::sink_minimum_batch_updates, but for storage persist_sink’s.

source

pub fn storage_source_decode_fuel(&self) -> usize

The maximum amount of work to do in the persist_source mfp_and_decode operator before yielding.

source

pub fn new_for_tests() -> Self

Returns a new instance of PersistConfig for tests.

source §

This initial value was tuned via a one-time experiment that showed an environment running our demo “auction” source + mv got 90%+ cache hits with a 1 MiB cache. This doesn’t scale up to prod data sizes and doesn’t help with multi-process replicas, but the memory usage seems unobjectionable enough to have it for the cases that it does help.

source