Expand description
Run-time configuration parameters
ยงOverview
Materialize roughly follows the PostgreSQL configuration model, which works
as follows. There is a global set of named configuration parameters, like
DateStyle
and client_encoding
. These parameters can be set in several
places: in an on-disk configuration file (in Postgres, named
postgresql.conf), in command line arguments when the server is started, or
at runtime via the ALTER SYSTEM
or SET
statements. Parameters that are
set in a session take precedence over database defaults, which in turn take
precedence over command line arguments, which in turn take precedence over
settings in the on-disk configuration. Note that changing the value of
parameters obeys transaction semantics: if a transaction fails to commit,
any parameters that were changed in that transaction (i.e., via SET
) will
be rolled back to their previous value.
The Materialize configuration hierarchy at the moment is much simpler. Global defaults are hardcoded into the binary, and a select few parameters can be overridden per session. A select few parameters can be overridden on disk.
The set of variables that can be overridden per session and the set of variables that can be overridden on disk are currently disjoint. The infrastructure has been designed with an eye towards merging these two sets and supporting additional layers to the hierarchy, however, should the need arise.
The configuration parameters that exist are driven by compatibility with PostgreSQL drivers that expect them, not because they are particularly important.
ยงStructure
Thw most meaningful exports from this module are:
-
SessionVars
represent per-session parameters, which each user can access independently of one another, and are accessed viaSET
.The fields of
SessionVars
are either;SessionVar
, which is preferable and simply requires full support of theSessionVar
impl for its embedded value type.ServerVar
for types that do not currently support everything required bySessionVar
, e.g. they are fixed-value parameters.
In the fullness of time, all fields in
SessionVars
should beSessionVar
. -
SystemVars
represent system-wide configuration settings and are accessed viaALTER SYSTEM SET
.All elements of
SystemVars
areSystemVar
.
Some VarDefinition
are also marked as a FeatureFlag
; this is just a
wrapper to make working with a set of VarDefinition
easier, primarily from
within SQL planning, where we might want to check if a feature is enabled
before planning it.
Modulesยง
- Configuration for how cluster replicas are scheduled.
- constraints ๐Defines constraints that can be imposed on variables.
- definitions ๐
- errors ๐
- Configuration for gRPC client connections.
- polyfill ๐The types in this module exist to overcome limitations of the Rust compiler, namely limitations in const generics.
- Tuning for RocksDB used by
UPSERT
sources that takes effect on restart. - value ๐
Structsยง
- This style should actually be some more complex struct, but we only support this configuration of it, so this is fine for the time being.
- flag.
- A
SessionVar
is the session value for a configuration parameter. If unset, the server default is used instead. - Session variables.
- A
SystemVar
is persisted on disk value for a configuration parameter. If unset, the server default is used instead. - On disk variables.
- Definition of a variable.
Enumsยง
- Severity levels can used to be used to filter which messages get sent to a client.
- The action to take during end_transaction.
- List of valid isolation levels.
- An owned version of
VarInput
. - List of valid time zones.
- The kinds of compiled in default values that can be used with
VarDefinition
. - Errors that can occur when working with
Var
s - Represents the input to a variable.
- Errors that can occur when parsing
VarInput
.
Constantsยง
- The name of the default database that Materialize uses.
- The major version of PostgreSQL that Materialize claims to be.
- The minor version of PostgreSQL that Materialize claims to be.
- The patch version of PostgreSQL that Materialize claims to be.
Staticsยง
- The default for the
DISK
option when creating managed clusters and cluster replicas. - Boolean flag indicating whether to enable syncing from LaunchDarkly. Can be turned off as an emergency measure to still be able to alter parameters while LD is broken.
- The interval we will fetch metadata from, unless overridden by the source.
- Controls the timeout when fetching kafka metadata. Defaults to 10s.
- Controls the timeout when fetching kafka progress records. Defaults to 60s.
- Controls
socket.connection.setup.timeout.ms
for rdkafka client connections. Defaults to the rdkafka default (30000ms). Cannot be greater thani32::MAX
or less than 1000ms - Enables
socket.keepalive.enable
for rdkafka client connections. Defaults to true. - Controls
socket.timeout.ms
for rdkafka client connections. Defaults to the rdkafka default (60000ms). Cannot be greater than 300000ms, more than 100ms greater thankafka_transaction_timeout
, or less than 10ms. - Controls
transaction.timeout.ms
for rdkafka client connections. Defaults to the rdkafka default (60000ms). Cannot be greater thani32::MAX
or less than 1000ms. - The logical compaction window for builtin tables and sources that have the
retained_metrics_relation
flag set. - Sets the
lock_wait_timeout
value to use during the snapshotting phase of MySQL sources. - Sets the
max_execution_time
value to use during the snapshotting phase of MySQL sources. - Sets the time between TCP keepalive probes when connecting to MySQL via
mz_mysql_util
. - Controls the connect_timeout setting when connecting to PG via
mz_postgres_util
. - Please see
PgSourceSnapshotConfig
. - Please see
PgSourceSnapshotConfig
. - Sets the
statement_timeout
value to use during the snapshotting phase of PG sources. - Please see
PgSourceSnapshotConfig
. - Sets whether to apply the TCP configuration parameters on the server when connecting to PG via
mz_postgres_util
. - Sets the amount of idle time before a keepalive packet is sent on the connection when connecting to PG via
mz_postgres_util
. - Sets the time interval between TCP keepalive probes when connecting to PG via
mz_postgres_util
. - Sets the maximum number of TCP keepalive probes that will be sent before dropping a connection when connecting to PG via
mz_postgres_util
. - Sets the TCP user timeout when connecting to PG via
mz_postgres_util
. - Sets the
wal_sender_timeout
value to use during the replication phase of PG sources. - Controls
mz_adapter::coord::timestamp_oracle::postgres_oracle::DynamicConfig::pg_connection_pool_max_size
. - Controls
mz_adapter::coord::timestamp_oracle::postgres_oracle::DynamicConfig::pg_connection_pool_max_wait
. - Controls
mz_adapter::coord::timestamp_oracle::postgres_oracle::DynamicConfig::pg_connection_pool_ttl
. - Controls
mz_adapter::coord::timestamp_oracle::postgres_oracle::DynamicConfig::pg_connection_pool_ttl_stagger
. - Feature flag indicating whether real time recency is enabled. Not that unlike other feature flags, this is made available at the session level, so is additionally gated by a feature flag.
- Controls the check interval for connections to SSH bastions via
mz_ssh_util
. - Controls the connect timeout for connections to SSH bastions via
mz_ssh_util
. - Controls the keepalive idle interval for connections to SSH bastions via
mz_ssh_util
. - The maximum number of in-flight bytes emitted by persist_sources feeding storage dataflows. Currently defaults to 256MiB = 268435456 bytes Note: Backpressure will only be turned on if disk is enabled based on
storage_dataflow_max_inflight_bytes_disk_only
flag - The fraction of the cluster replica size to be used as the maximum number of in-flight bytes emitted by persist_sources feeding storage dataflows. If not configured, the storage_dataflow_max_inflight_bytes value will be used. For this value to be used storage_dataflow_max_inflight_bytes needs to be set.
- Configuration ratio to shrink unusef buffers in upsert by. For eg: is 2 is set, then the buffers will be reduced by 2 i.e. halved. Default is 0, which means shrinking is disabled.
- The interval to collect statistics for
mz_source_statistics_per_worker
andmz_sink_statistics_per_worker
in clusterd. Controls the accuracy of metrics. - The interval to submit statistics to
mz_source_statistics_per_worker
andmz_sink_statistics_per_worker
.
Traitsยง
- Helper trait โฆ
- Defines a value that get stored as part of a System or Session variable.
- A
Var
represents a configuration parameter of an arbitrary type.
Functionsยง
- compat_
translate ๐If the given variable name and/or input is deprecated, return a corresponding updated value, otherwise return the original. - Returns whether the named variable is a cluster scheduling config
- Returns whether the named variable is an HTTP server related config var.
- Returns whether the named variable is a Postgres/CRDB timestamp oracle configuration parameter.
- Returns whether the named variable is a caching configuration parameter.