Struct dataflow_types::types::DataflowDescription [−][src]
pub struct DataflowDescription<View> {
pub source_imports: BTreeMap<GlobalId, SourceInstanceDesc>,
pub index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType)>,
pub objects_to_build: Vec<BuildDesc<View>>,
pub index_exports: Vec<(GlobalId, IndexDesc, RelationType)>,
pub sink_exports: Vec<(GlobalId, SinkDesc)>,
pub dependent_objects: BTreeMap<GlobalId, Vec<GlobalId>>,
pub as_of: Option<Antichain<Timestamp>>,
pub debug_name: String,
}
Expand description
A description of a dataflow to construct and results to surface.
Fields
source_imports: BTreeMap<GlobalId, SourceInstanceDesc>
Sources instantiations made available to the dataflow.
index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType)>
Indexes made available to the dataflow.
objects_to_build: Vec<BuildDesc<View>>
Views and indexes to be built and stored in the local context. Objects must be built in the specific order, as there may be dependencies of later objects on prior identifiers.
index_exports: Vec<(GlobalId, IndexDesc, RelationType)>
Indexes to be made available to be shared with other dataflows (id of new index, description of index, relationtype of base source/view)
sink_exports: Vec<(GlobalId, SinkDesc)>
sinks to be created (id of new sink, description of sink)
dependent_objects: BTreeMap<GlobalId, Vec<GlobalId>>
Maps views to views + indexes needed to generate that view
as_of: Option<Antichain<Timestamp>>
An optional frontier to which inputs should be advanced.
If this is set, it should override the default setting determined by
the upper bound of since
frontiers contributing to the dataflow.
It is an error for this to be set to a frontier not beyond that default.
debug_name: String
Human readable name
Implementations
pub fn import_index(
&mut self,
id: GlobalId,
description: IndexDesc,
typ: RelationType,
requesting_view: GlobalId
)
pub fn import_index(
&mut self,
id: GlobalId,
description: IndexDesc,
typ: RelationType,
requesting_view: GlobalId
)
Imports a previously exported index.
This method makes available an index previously exported as id
, identified
to the query by description
(which names the view the index arranges, and
the keys by which it is arranged).
The requesting_view
argument is currently necessary to correctly track the
dependencies of views on indexes.
pub fn import_source(
&mut self,
id: GlobalId,
description: SourceDesc,
persist: Option<SourcePersistDesc>
)
pub fn import_source(
&mut self,
id: GlobalId,
description: SourceDesc,
persist: Option<SourcePersistDesc>
)
Imports a source and makes it available as id
.
Binds to id
the relation expression view
.
Exports as id
an index on on_id
.
Future uses of import_index
in other dataflow descriptions may use id
,
as long as this dataflow has not been terminated in the meantime.
Exports as id
a sink described by description
.
Records a dependency of view_id
on depended_upon
.
Returns true iff id
is already imported.
Assigns the as_of
frontier to the supplied argument.
This method allows the dataflow to indicate a frontier up through which all times should be advanced. This can be done for at least two reasons: 1. correctness and 2. performance.
Correctness may require an as_of
to ensure that historical detail
is consolidated at representative times that do not present specific
detail that is not specifically correct. For example, updates may be
compacted to times that are no longer the source times, but instead
some byproduct of when compaction was executed; we should not present
those specific times as meaningfully different from other equivalent
times.
Performance may benefit from an aggressive as_of
as it reduces the
number of distinct moments at which collections vary. Differential
dataflow will refresh its outputs at each time its inputs change and
to moderate that we can minimize the volume of distinct input times
as much as possible.
Generally, one should consider setting as_of
at least to the since
frontiers of contributing data sources and as aggressively as the
computation permits.
Collects all indexes required to construct all exports.
Collects all transitively dependent identifiers that do not have their own dependencies.
Trait Implementations
Returns the “default value” for a type. Read more
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
Auto Trait Implementations
impl<View> RefUnwindSafe for DataflowDescription<View> where
View: RefUnwindSafe,
impl<View> Send for DataflowDescription<View> where
View: Send,
impl<View> Sync for DataflowDescription<View> where
View: Sync,
impl<View> Unpin for DataflowDescription<View> where
View: Unpin,
impl<View> UnwindSafe for DataflowDescription<View> where
View: UnwindSafe,
Blanket Implementations
Mutably borrows from an owned value. Read more
Attaches the provided Subscriber
to this type, returning a
WithDispatch
wrapper. Read more
Attaches the current default Subscriber
to this type, returning a
WithDispatch
wrapper. Read more