Module mz_storage_client::source ·
A controller that provides an interface to the storage layer.
The storage controller curates the creation of sources, the progress of readers through these collections, and their eventual dropping and resource reclamation.
The storage controller can be viewed as a partial map from
GlobalId to collection. It is an error to
use an identifier before it has been “created” with
create_source(). Once created, the controller holds
a read capability for each source, which is manipulated with
Eventually, the source is dropped with either
drop_sources() or by allowing compaction to the
- A tokio task (and support machinery) for maintaining storage-managed collections.
- Tracks persist shards ready to be finalized, i.e. remove the ability to read or write them. This identifies shards we no longer use, but did not have the opportunity to finalize before e.g. crashing.
- A tokio tasks (and support machinery) for dealing with the persist handles that the storage controller needs to hold.
- Rehydration of storage replicas.
- A tokio task (and support machinery) for producing storage statistics.
- Describes a request to create a source.
- Metadata required by a storage instance to read a storage collection
- State maintained about individual collections.
- A storage controller for a storage instance.
- Opaque token to ensure
prepare_exportis called before
create_exports. This token proves that compaction is being held back on
from_idat least until
idis created. It should be held while the AS OF is determined.
- The subset of
CollectionMetadatathat must be durable stored.
- State maintained about individual exports.
- A wrapper struct that presents the adapter token to a format that is understandable by persist and also allows us to differentiate between a token being present versus being set for the first time.
- Controller state maintained for each storage instance.
- Describes how data is written to the collection.
- Compaction policies for collections maintained by
- A trait that is used to calculate safe resumption frontiers for a source.