Struct mz_persist_client::PersistConfig
source · pub struct PersistConfig {Show 21 fields
pub(crate) build_version: Version,
pub now: NowFn,
pub blob_target_size: usize,
pub batch_builder_max_outstanding_parts: usize,
pub compaction_enabled: bool,
pub compaction_memory_bound_bytes: usize,
pub compaction_heuristic_min_inputs: usize,
pub compaction_heuristic_min_parts: usize,
pub compaction_heuristic_min_updates: usize,
pub compaction_concurrency_limit: usize,
pub compaction_queue_size: usize,
pub gc_batch_part_delete_concurrency_limit: usize,
pub compaction_minimum_timeout: Duration,
pub consensus_connection_pool_max_size: usize,
pub consensus_connection_pool_ttl: Duration,
pub consensus_connection_pool_ttl_stagger: Duration,
pub state_versions_recent_live_diffs_limit: usize,
pub writer_lease_duration: Duration,
pub reader_lease_duration: Duration,
pub critical_downgrade_interval: Duration,
pub hostname: String,
}
Expand description
The tunable knobs for persist.
Fields§
§build_version: Version
Info about which version of the code is running.
now: NowFn
A clock to use for all leasing and other non-debugging use.
blob_target_size: usize
A target maximum size of blob payloads in bytes. If a logical “batch” is bigger than this, it will be broken up into smaller, independent pieces. This is best-effort, not a guarantee (though as of 2022-06-09, we happen to always respect it). This target size doesn’t apply for an individual update that exceeds it in size, but that scenario is almost certainly a mis-use of the system.
batch_builder_max_outstanding_parts: usize
The maximum number of parts (s3 blobs) that crate::batch::BatchBuilder will pipeline before back-pressuring crate::batch::BatchBuilder::add calls on previous ones finishing.
compaction_enabled: bool
Whether to physically and logically compact batches in blob storage.
compaction_memory_bound_bytes: usize
The upper bound on compaction’s memory consumption. The value must be at
least 4*blob_target_size
. Increasing this value beyond the minimum allows
compaction to merge together more runs at once, providing greater
consolidation of updates, at the cost of greater memory usage.
compaction_heuristic_min_inputs: usize
In Compactor::compact_and_apply, we do the compaction (don’t skip it) if the number of inputs is at least this many. Compaction is performed if any of the heuristic criteria are met (they are OR’d).
compaction_heuristic_min_parts: usize
In Compactor::compact_and_apply, we do the compaction (don’t skip it) if the number of batch parts is at least this many. Compaction is performed if any of the heuristic criteria are met (they are OR’d).
compaction_heuristic_min_updates: usize
In Compactor::compact_and_apply, we do the compaction (don’t skip it) if the number of updates is at least this many. Compaction is performed if any of the heuristic criteria are met (they are OR’d).
compaction_concurrency_limit: usize
In Compactor::compact_and_apply_background, the maximum number of concurrent compaction requests that can execute for a given shard.
compaction_queue_size: usize
In Compactor::compact_and_apply_background, the maximum number of pending compaction requests to queue.
gc_batch_part_delete_concurrency_limit: usize
The maximum number of concurrent blob deletes during garbage collection.
compaction_minimum_timeout: Duration
In Compactor::compact_and_apply_background, the minimum amount of time to allow a compaction request to run before timing it out. A request may be given a timeout greater than this value depending on the inputs’ size
consensus_connection_pool_max_size: usize
The maximum size of the connection pool to Postgres/CRDB when performing consensus reads and writes.
consensus_connection_pool_ttl: Duration
The minimum TTL of a connection to Postgres/CRDB before it is proactively terminated. Connections are routinely culled to balance load against the downstream database.
consensus_connection_pool_ttl_stagger: Duration
The minimum time between TTLing connections to Postgres/CRDB. This delay is
used to stagger reconnections to avoid stampedes and high tail latencies.
This value should be much less than consensus_connection_pool_ttl
so that
reconnections are biased towards terminating the oldest connections first.
A value of consensus_connection_pool_ttl / consensus_connection_pool_max_size
is likely a good place to start so that all connections are rotated when the
pool is fully used.
state_versions_recent_live_diffs_limit: usize
The # of diffs to initially scan when fetching the latest consensus state, to determine which requests go down the fast vs slow path. Should be large enough to fetch all live diffs in the steady-state, and small enough to query Consensus at high volume. Steady-state usage should accommodate readers that require seqno-holds for reasonable amounts of time, which to start we say is 10s of minutes.
This value ought to be defined in terms of NEED_ROLLUP_THRESHOLD
to approximate
when we expect rollups to be written and therefore when old states will be truncated
by GC.
writer_lease_duration: Duration
Length of time after a writer’s last operation after which the writer may be expired.
reader_lease_duration: Duration
Length of time after a reader’s last operation after which the reader may be expired.
critical_downgrade_interval: Duration
Length of time between critical handles’ calls to downgrade since
hostname: String
Hostname of this persist user. Stored in state and used for debugging.
Implementations§
source§impl PersistConfig
impl PersistConfig
sourcepub fn new(build_info: &BuildInfo, now: NowFn) -> Self
pub fn new(build_info: &BuildInfo, now: NowFn) -> Self
Returns a new instance of PersistConfig with default tuning.
source§impl PersistConfig
impl PersistConfig
sourcepub const DEFAULT_BLOB_TARGET_SIZE: usize = 134_217_728usize
pub const DEFAULT_BLOB_TARGET_SIZE: usize = 134_217_728usize
Default value for PersistConfig::blob_target_size
.
sourcepub const DEFAULT_COMPACTION_MINIMUM_TIMEOUT: Duration = _
pub const DEFAULT_COMPACTION_MINIMUM_TIMEOUT: Duration = _
Default value for PersistConfig::compaction_minimum_timeout
.
pub(crate) const DEFAULT_READ_LEASE_DURATION: Duration = _
pub(crate) const NEED_ROLLUP_THRESHOLD: u64 = 128u64
Trait Implementations§
source§impl Clone for PersistConfig
impl Clone for PersistConfig
source§fn clone(&self) -> PersistConfig
fn clone(&self) -> PersistConfig
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moresource§impl ConsensusKnobs for PersistConfig
impl ConsensusKnobs for PersistConfig
source§fn connection_pool_max_size(&self) -> usize
fn connection_pool_max_size(&self) -> usize
source§fn connection_pool_ttl(&self) -> Duration
fn connection_pool_ttl(&self) -> Duration
source§fn connection_pool_ttl_stagger(&self) -> Duration
fn connection_pool_ttl_stagger(&self) -> Duration
Auto Trait Implementations§
impl !RefUnwindSafe for PersistConfig
impl Send for PersistConfig
impl Sync for PersistConfig
impl Unpin for PersistConfig
impl !UnwindSafe for PersistConfig
Blanket Implementations§
source§impl<T> FutureExt for T
impl<T> FutureExt for T
source§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
source§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
source§impl<T> Instrument for T
impl<T> Instrument for T
source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
source§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T
in a tonic::Request
source§impl<P, R> ProtoType<R> for Pwhere
R: RustType<P>,
impl<P, R> ProtoType<R> for Pwhere
R: RustType<P>,
source§fn into_rust(self) -> Result<R, TryFromProtoError>
fn into_rust(self) -> Result<R, TryFromProtoError>
RustType::from_proto
.source§fn from_rust(rust: &R) -> P
fn from_rust(rust: &R) -> P
RustType::into_proto
.