Expand description
Dyncfgs used by the storage layer. Despite their name, these can be used “statically” during rendering, or dynamically within timely operators.
Constants§
- CLUSTER_
SHUTDOWN_ GRACE_ PERIOD - When dataflows observe an invariant violation it is either due to a bug or due to the cluster being shut down. This configuration defines the amount of time to wait before panicking the process, which will register the invariant violation.
- DELAY_
SOURCES_ PAST_ REHYDRATION - Whether rendering should use
mz_join_core
rather than DD’sJoinCore::join_core
. Configuration for basic hydration backpressure. - ENFORCE_
EXTERNAL_ ADDRESSES - Whether or not to enforce that external connection addresses are global (not private or local) when resolving them.
- KAFKA_
BUFFERED_ EVENT_ RESIZE_ THRESHOLD_ ELEMENTS - KAFKA_
CLIENT_ ID_ ENRICHMENT_ RULES - Rules for enriching the
client.id
property of Kafka clients with additional data. - KAFKA_
DEFAULT_ AWS_ PRIVATELINK_ ENDPOINT_ IDENTIFICATION_ ALGORITHM - KAFKA_
POLL_ MAX_ WAIT - The maximum time we will wait before re-polling rdkafka to see if new partitions/data are available.
- MYSQL_
OFFSET_ KNOWN_ INTERVAL - Interval to fetch
offset_known
, from@gtid_executed
- MYSQL_
REPLICATION_ HEARTBEAT_ INTERVAL - Replication heartbeat interval requested from the MySQL server.
- ORE_
OVERFLOWING_ BEHAVIOR - Configure mz-ore overflowing type behavior.
- PG_
FETCH_ SLOT_ RESUME_ LSN_ INTERVAL - Interval to poll
confirmed_flush_lsn
to get a resumption lsn. - PG_
OFFSET_ KNOWN_ INTERVAL - Interval to fetch
offset_known
, frompg_current_wal_lsn
- PG_
SCHEMA_ VALIDATION_ INTERVAL - Interval to re-validate the schemas of ingested tables.
- REPLICA_
METRICS_ HISTORY_ RETENTION_ INTERVAL - The interval of time to keep when truncating the replica metrics history.
- SINK_
ENSURE_ TOPIC_ CONFIG - Configure how to behave when trying to create an existing topic with specified configs.
- SINK_
PROGRESS_ SEARCH - If set, iteratively search the progress topic for a progress record with increasing lookback.
- STATISTICS_
RETENTION_ DURATION - The time after which we delete per-replica statistics (for sources and sinks) after there have been no updates.
- STORAGE_
DOWNGRADE_ SINCE_ DURING_ FINALIZATION - When enabled, force-downgrade the controller’s since handle on the shard during shard finalization.
- STORAGE_
RECLOCK_ TO_ LATEST - Whether to mint reclock bindings based on the latest probed frontier or the currently ingested frontier.
- STORAGE_
ROCKSDB_ CLEANUP_ TRIES - How many times to try to cleanup old RocksDB DB’s on disk before giving up.
- STORAGE_
ROCKSDB_ USE_ MERGE_ OPERATOR - Whether to enable the merge operator in upsert for the RocksDB backend.
- STORAGE_
SERVER_ MAINTENANCE_ INTERVAL - The interval at which the storage server performs maintenance tasks.
- STORAGE_
SINK_ SNAPSHOT_ FRONTIER - If true, skip fetching the snapshot in the sink once the frontier has advanced.
- STORAGE_
SUSPEND_ AND_ RESTART_ DELAY - Delay interval when reconnecting to a source / sink after halt.
- STORAGE_
UPSERT_ MAX_ SNAPSHOT_ BATCH_ BUFFERING - If
storage_upsert_prevent_snapshot_buffering
is true, this prevents the upsert operator from buffering too many events from the upstream snapshot. In the absence of hydration flow control, this could prevent certain workloads from causing egregiously large writes to RocksDB. - STORAGE_
UPSERT_ PREVENT_ SNAPSHOT_ BUFFERING - Whether or not to prevent buffering the entire upstream snapshot in memory when processing it in memory. This is generally understood to reduce memory consumption.
- STORAGE_
USE_ CONTINUAL_ FEEDBACK_ UPSERT - Whether to use the new continual feedback upsert operator.
- SUSPENDABLE_
SOURCES - Whether storage dataflows should suspend execution while downstream operators are still processing data.
- WALLCLOCK_
GLOBAL_ LAG_ HISTOGRAM_ RETENTION_ INTERVAL - The interval of time to keep when truncating the wallclock lag histogram.
- WALLCLOCK_
LAG_ HISTORY_ RETENTION_ INTERVAL - The interval of time to keep when truncating the wallclock lag history.
Statics§
- KAFKA_
METADATA_ FETCH_ INTERVAL - Interval to fetch topic partition metadata.
Functions§
- all_
dyncfgs - Adds the full set of all storage
Config
s.