Struct dataflow::render::StorageState

pub struct StorageState {
    pub local_inputs: HashMap<GlobalId, LocalInput>,
    pub source_descriptions: HashMap<GlobalId, SourceDesc>,
    pub ts_source_mapping: HashMap<GlobalId, Vec<Weak<Option<SourceToken>>>>,
    pub ts_histories: HashMap<GlobalId, TimestampBindingRc>,
    pub persisted_sources: PersistedSourceManager,
    pub metrics: Metrics,
    pub persist: Option<RuntimeClient>,
}

Worker-local state related to the ingress or egress of collections of data.

Fields

local_inputs: HashMap<GlobalId, LocalInput>

Handles to local inputs, keyed by ID.

source_descriptions: HashMap<GlobalId, SourceDesc>

Source descriptions that have been created and not yet dropped.

For the moment we retain all source descriptions, even those that have been dropped, as this is used to check for rebinding of previous identifiers. Once we have a better mechanism to avoid that, for example that identifiers must strictly increase, we can clean up descriptions when sources are dropped.

ts_source_mapping: HashMap<GlobalId, Vec<Weak<Option<SourceToken>>>>

Handles to external sources, keyed by ID.

ts_histories: HashMap<GlobalId, TimestampBindingRc>

Timestamp data updates for each source.

persisted_sources: PersistedSourceManager

Handles that allow setting the compaction frontier for a persisted source. There can only ever be one running (rendered) source of a persisted source, and if there is one, this map will contain a handle to it.

metrics: Metrics

Metrics reported by all dataflows.

persist: Option<RuntimeClient>

Handle to the persistence runtime. None if disabled.