Struct mz_storage::storage_state::StorageState

source · 
pub struct StorageState {
Show 24 fields
    pub source_uppers: BTreeMap<GlobalId, Rc<RefCell<Antichain<Timestamp>>>>,
    pub source_tokens: BTreeMap<GlobalId, Vec<PressOnDropButton>>,
    pub metrics: StorageMetrics,
    pub reported_frontiers: BTreeMap<GlobalId, Antichain<Timestamp>>,
    pub ingestions: BTreeMap<GlobalId, IngestionDescription<CollectionMetadata>>,
    pub exports: BTreeMap<GlobalId, StorageSinkDesc<MetadataFilled, Timestamp>>,
    pub now: NowFn,
    pub timely_worker_index: usize,
    pub timely_worker_peers: usize,
    pub instance_context: StorageInstanceContext,
    pub persist_clients: Arc<PersistClientCache>,
    pub txns_ctx: TxnsContext,
    pub sink_tokens: BTreeMap<GlobalId, Vec<PressOnDropButton>>,
    pub sink_write_frontiers: BTreeMap<GlobalId, Rc<RefCell<Antichain<Timestamp>>>>,
    pub dropped_ids: BTreeSet<GlobalId>,
    pub aggregated_statistics: AggregatedStatistics,
    pub object_status_updates: Rc<RefCell<Vec<StatusUpdate>>>,
    pub internal_cmd_tx: Rc<RefCell<dyn InternalCommandSender>>,
    pub read_only_rx: Receiver<bool>,
    pub read_only_tx: Sender<bool>,
    pub async_worker: AsyncStorageWorker<Timestamp>,
    pub storage_configuration: StorageConfiguration,
    pub dataflow_parameters: DataflowParameters,
    pub tracing_handle: Arc<TracingHandle>,

}
Expand description

Worker-local state related to the ingress or egress of collections of data.

Fields§

§source_uppers: BTreeMap<GlobalId, Rc<RefCell<Antichain<Timestamp>>>>

The highest observed upper frontier for collection.

This is shared among all source instances, so that they can jointly advance the frontier even as other instances are created and dropped. Ideally, the Storage module would eventually provide one source of truth on this rather than multiple, and we should aim for that but are not there yet.

§source_tokens: BTreeMap<GlobalId, Vec<PressOnDropButton>>

Handles to created sources, keyed by ID NB: The type of the tokens must not be changed to something other than PressOnDropButton to prevent usage of custom shutdown tokens that are tricky to get right.

§metrics: StorageMetrics

Metrics for storage objects.

§reported_frontiers: BTreeMap<GlobalId, Antichain<Timestamp>>

Tracks the conditional write frontiers we have reported.

§ingestions: BTreeMap<GlobalId, IngestionDescription<CollectionMetadata>>

Descriptions of each installed ingestion.

§exports: BTreeMap<GlobalId, StorageSinkDesc<MetadataFilled, Timestamp>>

Descriptions of each installed export.

§now: NowFn

Undocumented

§timely_worker_index: usize

Index of the associated timely dataflow worker.

§timely_worker_peers: usize

Peers in the associated timely dataflow worker.

§instance_context: StorageInstanceContext

Other configuration for sources and sinks.

§persist_clients: Arc<PersistClientCache>

A process-global cache of (blob_uri, consensus_uri) -> PersistClient. This is intentionally shared between workers

§txns_ctx: TxnsContext

Context necessary for rendering txn-wal operators.

§sink_tokens: BTreeMap<GlobalId, Vec<PressOnDropButton>>

Tokens that should be dropped when a dataflow is dropped to clean up associated state. NB: The type of the tokens must not be changed to something other than PressOnDropButton to prevent usage of custom shutdown tokens that are tricky to get right.

§sink_write_frontiers: BTreeMap<GlobalId, Rc<RefCell<Antichain<Timestamp>>>>

Frontier of sink writes (all subsequent writes will be at times at or equal to this frontier)

§dropped_ids: BTreeSet<GlobalId>

Collection ids that have been dropped but not yet reported as dropped

§aggregated_statistics: AggregatedStatistics

Statistics for sources and sinks.

§object_status_updates: Rc<RefCell<Vec<StatusUpdate>>>

Status updates reported by health operators.

NOTE: Operators that append to this collection should take care to only add new status updates if the status of the ingestion/export in question has changed.

§internal_cmd_tx: Rc<RefCell<dyn InternalCommandSender>>

Sender for cluster-internal storage commands. These can be sent from within workers/operators and will be distributed to all workers. For example, for shutting down an entire dataflow from within a operator/worker.

§read_only_rx: Receiver<bool>

When this replica/cluster is in read-only mode it must not affect any changes to external state. This flag can only be changed by a StorageCommand::AllowWrites.

Everything running on this replica/cluster must obey this flag. At the time of writing, nothing currently looks at this flag. TODO(benesch): fix this.

NOTE: In the future, we might want a more complicated flag, for example something that tells us after which timestamp we are allowed to write. In this first version we are keeping things as simple as possible!

§read_only_tx: Sender<bool>

Send-side for read-only state.

§async_worker: AsyncStorageWorker<Timestamp>

Async worker companion, used for running code that requires async, which the timely main loop cannot do.

§storage_configuration: StorageConfiguration

Configuration for source and sink connections.

§dataflow_parameters: DataflowParameters

Dynamically configurable parameters that control how dataflows are rendered. NOTE(guswynn): we should consider moving these into storage_configuration.

§tracing_handle: Arc<TracingHandle>

A process-global handle to tracing configuration.

Implementations§

source§

impl StorageState

source

pub fn handle_storage_command(&mut self, cmd: StorageCommand)

Entry point for applying a storage command.

NOTE: This does not have access to the timely worker and therefore cannot render dataflows. For dataflow rendering, this needs to either send asynchronous command to the async_worker or internal commands to the internal_cmd_tx.

source

fn drop_collection(&mut self, id: GlobalId)

Drop the identified storage collection from the storage state.

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T, U> CastInto<U> for T
where U: CastFrom<T>,

source§

fn cast_into(self) -> U

Performs the cast.
source§

impl<T> Conv for T

source§

fn conv<T>(self) -> T
where Self: Into<T>,

Converts self into T using Into<T>. Read more
source§

impl<T> FmtForward for T

source§

fn fmt_binary(self) -> FmtBinary<Self>
where Self: Binary,

Causes self to use its Binary implementation when Debug-formatted.
source§

fn fmt_display(self) -> FmtDisplay<Self>
where Self: Display,

Causes self to use its Display implementation when Debug-formatted.
source§

fn fmt_lower_exp(self) -> FmtLowerExp<Self>
where Self: LowerExp,

Causes self to use its LowerExp implementation when Debug-formatted.
source§

fn fmt_lower_hex(self) -> FmtLowerHex<Self>
where Self: LowerHex,

Causes self to use its LowerHex implementation when Debug-formatted.
source§

fn fmt_octal(self) -> FmtOctal<Self>
where Self: Octal,

Causes self to use its Octal implementation when Debug-formatted.
source§

fn fmt_pointer(self) -> FmtPointer<Self>
where Self: Pointer,

Causes self to use its Pointer implementation when Debug-formatted.
source§

fn fmt_upper_exp(self) -> FmtUpperExp<Self>
where Self: UpperExp,

Causes self to use its UpperExp implementation when Debug-formatted.
source§

fn fmt_upper_hex(self) -> FmtUpperHex<Self>
where Self: UpperHex,

Causes self to use its UpperHex implementation when Debug-formatted.
source§

fn fmt_list(self) -> FmtList<Self>
where &'a Self: for<'a> IntoIterator,

Formats each item in a sequence. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T> FutureExt for T

source§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
source§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
source§

impl<T> Instrument for T

source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> IntoRequest<T> for T

source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
source§

impl<Unshared, Shared> IntoShared<Shared> for Unshared
where Shared: FromUnshared<Unshared>,

source§

fn into_shared(self) -> Shared

Creates a shared type from an unshared type.
source§

impl<T, U> OverrideFrom<Option<&T>> for U
where U: OverrideFrom<T>,

source§

fn override_from(self, layer: &Option<&T>) -> U

Override the configuration represented by Self with values from the given layer.
source§

impl<T> Pipe for T
where T: ?Sized,

source§

fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> R
where Self: Sized,

Pipes by value. This is generally the method you want to use. Read more
source§

fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> R
where R: 'a,

Borrows self and passes that borrow into the pipe function. Read more
source§

fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> R
where R: 'a,

Mutably borrows self and passes that borrow into the pipe function. Read more
source§

fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
where Self: Borrow<B>, B: 'a + ?Sized, R: 'a,

Borrows self, then passes self.borrow() into the pipe function. Read more
source§

fn pipe_borrow_mut<'a, B, R>( &'a mut self, func: impl FnOnce(&'a mut B) -> R, ) -> R
where Self: BorrowMut<B>, B: 'a + ?Sized, R: 'a,

Mutably borrows self, then passes self.borrow_mut() into the pipe function. Read more
source§

fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
where Self: AsRef<U>, U: 'a + ?Sized, R: 'a,

Borrows self, then passes self.as_ref() into the pipe function.
source§

fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
where Self: AsMut<U>, U: 'a + ?Sized, R: 'a,

Mutably borrows self, then passes self.as_mut() into the pipe function.
source§

fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
where Self: Deref<Target = T>, T: 'a + ?Sized, R: 'a,

Borrows self, then passes self.deref() into the pipe function.
source§

fn pipe_deref_mut<'a, T, R>( &'a mut self, func: impl FnOnce(&'a mut T) -> R, ) -> R
where Self: DerefMut<Target = T> + Deref, T: 'a + ?Sized, R: 'a,

Mutably borrows self, then passes self.deref_mut() into the pipe function.
source§

impl<T> Pointable for T

source§

const ALIGN: usize = _

The alignment of pointer.
§

type Init = T

The type for initializers.
source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
source§

impl<P, R> ProtoType<R> for P
where R: RustType<P>,

source§

fn into_rust(self) -> Result<R, TryFromProtoError>

See RustType::from_proto.
source§

fn from_rust(rust: &R) -> P

See RustType::into_proto.
source§

impl<T> Same for T

§

type Output = T

Should always be Self
source§

impl<'a, S, T> Semigroup<&'a S> for T
where T: Semigroup<S>,

source§

fn plus_equals(&mut self, rhs: &&'a S)

The method of std::ops::AddAssign, for types that do not implement AddAssign.
source§

impl<T> Tap for T

source§

fn tap(self, func: impl FnOnce(&Self)) -> Self

Immutable access to a value. Read more
source§

fn tap_mut(self, func: impl FnOnce(&mut Self)) -> Self

Mutable access to a value. Read more
source§

fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
where Self: Borrow<B>, B: ?Sized,

Immutable access to the Borrow<B> of a value. Read more
source§

fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
where Self: BorrowMut<B>, B: ?Sized,

Mutable access to the BorrowMut<B> of a value. Read more
source§

fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
where Self: AsRef<R>, R: ?Sized,

Immutable access to the AsRef<R> view of a value. Read more
source§

fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
where Self: AsMut<R>, R: ?Sized,

Mutable access to the AsMut<R> view of a value. Read more
source§

fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
where Self: Deref<Target = T>, T: ?Sized,

Immutable access to the Deref::Target of a value. Read more
source§

fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
where Self: DerefMut<Target = T> + Deref, T: ?Sized,

Mutable access to the Deref::Target of a value. Read more
source§

fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self

Calls .tap() only in debug builds, and is erased in release builds.
source§

fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self

Calls .tap_mut() only in debug builds, and is erased in release builds.
source§

fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
where Self: Borrow<B>, B: ?Sized,

Calls .tap_borrow() only in debug builds, and is erased in release builds.
source§

fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
where Self: BorrowMut<B>, B: ?Sized,

Calls .tap_borrow_mut() only in debug builds, and is erased in release builds.
source§

fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
where Self: AsRef<R>, R: ?Sized,

Calls .tap_ref() only in debug builds, and is erased in release builds.
source§

fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
where Self: AsMut<R>, R: ?Sized,

Calls .tap_ref_mut() only in debug builds, and is erased in release builds.
source§

fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
where Self: Deref<Target = T>, T: ?Sized,

Calls .tap_deref() only in debug builds, and is erased in release builds.
source§

fn tap_deref_mut_dbg<T>(self, func: impl FnOnce(&mut T)) -> Self
where Self: DerefMut<Target = T> + Deref, T: ?Sized,

Calls .tap_deref_mut() only in debug builds, and is erased in release builds.
source§

impl<T> TryConv for T

source§

fn try_conv<T>(self) -> Result<T, Self::Error>
where Self: TryInto<T>,

Attempts to convert self into T using TryInto<T>. Read more
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

source§

fn vzip(self) -> V

source§

impl<T> WithSubscriber for T

source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more