Module mz_storage_controller::collection_mgmt

Tokio tasks (and support machinery) for maintaining storage-managed collections.

We differentiate between append-only collections and differential collections. The intent is that knowing the type allows being more intentional about what state we keep in memory and how we work when in read-only mode / during zero-downtime upgrades.

Append-only collections

Writers only append blind writes. Those writes never fail. It does not matter at what timestamp they happen (to a degree, but …).

While in read-only mode, the append-only write task can immediately write updates out as batches, but only append them when going out of read-only mode.

Differential collections

These are very similar to the self-correcting persist_sink. We have an in-memory desired state and continually make it so that persist matches desired. As described below (in the task implementation), we could do this in a memory efficient way by keeping open a persist read handle and continually updating/consolidating our desired collection. This way, we would be memory-efficient even in read-only mode.

This is an evolution of the current design where, on startup, we bring the persist collection into a known state (mostly by retracting everything) and then assume that this envd is the only writer. We panic when that is ever not the case, which we notice when the upper of a collection changes unexpectedly. With this new design we can instead continually update our view of the persist shard and emit updates when needed, when desired changed.

NOTE: As it is, we always keep all of desired in memory. Only when told to go out of read-only mode would we start attempting to write.

Read-only mode

When CollectionManager is in read-only mode it cannot write out to persist. It will, however, maintain the desired state of differential collections so that we can immediately start writing out updates when going out of read-only mode.

For append-only collections we either panic, in the case of CollectionManager::blind_write, or report back a StorageError::ReadOnly when trying to append through a MonotonicAppender returned from CollectionManager::monotonic_appender.

Structs

• CollectionManager
• DifferentialWriteTask 🔒
  A task that will make it so that the state in persist matches the desired state and continuously bump the upper for the specified collection.

Constants

• DEFAULT_TICK_MS 🔒

Functions

• append_only_write_task 🔒
  Spawns an mz_ore::task that will continuously bump the upper for the specified collection, and append data that is sent via the provided mpsc::UnboundedSender.
• monotonic_append 🔒
• notify_listeners 🔒

Type Aliases

• AppendOnlyWriteChannel 🔒
  A channel for sending writes to an append-only collection.
• DifferentialWriteChannel 🔒
  A channel for sending writes to a differential collection.
• ShutdownSender 🔒
• WriteTask 🔒