Module mz_storage_controller::collection_mgmt
source · Expand description
Tokio tasks (and support machinery) for maintaining storage-managed collections.
We differentiate between append-only collections and differential collections. The intent is that knowing the type allows being more intentional about what state we keep in memory and how we work when in read-only mode / during zero-downtime upgrades.
§Append-only collections
Writers only append blind writes. Those writes never fail. It does not matter at what timestamp they happen (to a degree, but …).
While in read-only mode, the append-only write task can immediately write updates out as batches, but only append them when going out of read-only mode.
§Differential collections
These are very similar to the self-correcting persist_sink. We have an in-memory desired state and continually make it so that persist matches desired. As described below (in the task implementation), we could do this in a memory efficient way by keeping open a persist read handle and continually updating/consolidating our desired collection. This way, we would be memory-efficient even in read-only mode.
This is an evolution of the current design where, on startup, we bring the
persist collection into a known state (mostly by retracting everything) and
then assume that this envd
is the only writer. We panic when that is ever
not the case, which we notice when the upper of a collection changes
unexpectedly. With this new design we can instead continually update our
view of the persist shard and emit updates when needed, when desired
changed.
NOTE: As it is, we always keep all of desired in memory. Only when told to go out of read-only mode would we start attempting to write.
§Read-only mode
When CollectionManager
is in read-only mode it cannot write out to
persist. It will, however, maintain the desired
state of differential
collections so that we can immediately start writing out updates when going
out of read-only mode.
For append-only collections we either panic, in the case of
CollectionManager::blind_write
, or report back a
StorageError::ReadOnly
when trying to append through a
MonotonicAppender
returned from
CollectionManager::monotonic_appender
.
Structs§
- A task that will make it so that the state in persist matches the desired state and continuously bump the upper for the specified collection.
Constants§
Functions§
- Spawns an
mz_ore::task
that will continuously bump the upper for the specified collection, and append data that is sent via the providedmpsc::UnboundedSender
.
Type Aliases§
- A channel for sending writes to an append-only collection.
- A channel for sending writes to a differential collection.