Struct mz_compute_client::command::DataflowDescription
source · [−]pub struct DataflowDescription<P, S = (), T = Timestamp> {
pub source_imports: BTreeMap<GlobalId, (SourceInstanceDesc<S>, bool)>,
pub index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType, bool)>,
pub objects_to_build: Vec<BuildDesc<P>>,
pub index_exports: BTreeMap<GlobalId, (IndexDesc, RelationType)>,
pub sink_exports: BTreeMap<GlobalId, ComputeSinkDesc<S, T>>,
pub as_of: Option<Antichain<T>>,
pub until: Antichain<T>,
pub debug_name: String,
}
Expand description
A description of a dataflow to construct and results to surface.
Fields
source_imports: BTreeMap<GlobalId, (SourceInstanceDesc<S>, bool)>
Sources instantiations made available to the dataflow pair with monotonicity information.
index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType, bool)>
Indexes made available to the dataflow. (id of new index, description of index, relationtype of base source/view, monotonic)
objects_to_build: Vec<BuildDesc<P>>
Views and indexes to be built and stored in the local context. Objects must be built in the specific order, as there may be dependencies of later objects on prior identifiers.
index_exports: BTreeMap<GlobalId, (IndexDesc, RelationType)>
Indexes to be made available to be shared with other dataflows (id of new index, description of index, relationtype of base source/view)
sink_exports: BTreeMap<GlobalId, ComputeSinkDesc<S, T>>
sinks to be created (id of new sink, description of sink)
as_of: Option<Antichain<T>>
An optional frontier to which inputs should be advanced.
If this is set, it should override the default setting determined by
the upper bound of since
frontiers contributing to the dataflow.
It is an error for this to be set to a frontier not beyond that default.
until: Antichain<T>
Frontier beyond which the dataflow should not execute.
Specifically, updates at times greater or equal to this frontier are suppressed.
This is often set to as_of + 1
to enable “batch” computations.
debug_name: String
Human readable name
Implementations
sourceimpl<T> DataflowDescription<OptimizedMirRelationExpr, (), T>
impl<T> DataflowDescription<OptimizedMirRelationExpr, (), T>
sourcepub fn new(name: String) -> Self
pub fn new(name: String) -> Self
Creates a new dataflow description with a human-readable name.
sourcepub fn import_index(
&mut self,
id: GlobalId,
description: IndexDesc,
typ: RelationType,
monotonic: bool
)
pub fn import_index(
&mut self,
id: GlobalId,
description: IndexDesc,
typ: RelationType,
monotonic: bool
)
Imports a previously exported index.
This method makes available an index previously exported as id
, identified
to the query by description
(which names the view the index arranges, and
the keys by which it is arranged).
sourcepub fn import_source(&mut self, id: GlobalId, typ: RelationType, monotonic: bool)
pub fn import_source(&mut self, id: GlobalId, typ: RelationType, monotonic: bool)
Imports a source and makes it available as id
.
sourcepub fn insert_plan(&mut self, id: GlobalId, plan: OptimizedMirRelationExpr)
pub fn insert_plan(&mut self, id: GlobalId, plan: OptimizedMirRelationExpr)
Binds to id
the relation expression plan
.
sourcepub fn export_index(
&mut self,
id: GlobalId,
description: IndexDesc,
on_type: RelationType
)
pub fn export_index(
&mut self,
id: GlobalId,
description: IndexDesc,
on_type: RelationType
)
Exports as id
an index described by description
.
Future uses of import_index
in other dataflow descriptions may use id
,
as long as this dataflow has not been terminated in the meantime.
sourcepub fn export_sink(&mut self, id: GlobalId, description: ComputeSinkDesc<(), T>)
pub fn export_sink(&mut self, id: GlobalId, description: ComputeSinkDesc<(), T>)
Exports as id
a sink described by description
.
sourcepub fn is_imported(&self, id: &GlobalId) -> bool
pub fn is_imported(&self, id: &GlobalId) -> bool
Returns true iff id
is already imported.
sourcepub fn set_as_of(&mut self, as_of: Antichain<T>)
pub fn set_as_of(&mut self, as_of: Antichain<T>)
Assigns the as_of
frontier to the supplied argument.
This method allows the dataflow to indicate a frontier up through which all times should be advanced. This can be done for at least two reasons: 1. correctness and 2. performance.
Correctness may require an as_of
to ensure that historical detail
is consolidated at representative times that do not present specific
detail that is not specifically correct. For example, updates may be
compacted to times that are no longer the source times, but instead
some byproduct of when compaction was executed; we should not present
those specific times as meaningfully different from other equivalent
times.
Performance may benefit from an aggressive as_of
as it reduces the
number of distinct moments at which collections vary. Differential
dataflow will refresh its outputs at each time its inputs change and
to moderate that we can minimize the volume of distinct input times
as much as possible.
Generally, one should consider setting as_of
at least to the since
frontiers of contributing data sources and as aggressively as the
computation permits.
sourceimpl<P, S, T> DataflowDescription<P, S, T>where
P: CollectionPlan,
impl<P, S, T> DataflowDescription<P, S, T>where
P: CollectionPlan,
sourcepub fn export_ids(&self) -> impl Iterator<Item = GlobalId> + '_
pub fn export_ids(&self) -> impl Iterator<Item = GlobalId> + '_
Identifiers of exported objects (indexes and sinks).
sourcepub fn subscribe_ids(&self) -> impl Iterator<Item = GlobalId> + '_
pub fn subscribe_ids(&self) -> impl Iterator<Item = GlobalId> + '_
Identifiers of exported subscribe sinks.
sourcepub fn build_desc(&self, id: GlobalId) -> &BuildDesc<P>
pub fn build_desc(&self, id: GlobalId) -> &BuildDesc<P>
Returns the description of the object to build with the specified identifier.
Panics
Panics if id
is not present in objects_to_build
exactly once.
sourcepub fn depends_on(&self, collection_id: GlobalId) -> BTreeSet<GlobalId>
pub fn depends_on(&self, collection_id: GlobalId) -> BTreeSet<GlobalId>
Computes the set of identifiers upon which the specified collection identifier depends.
id
must specify a valid object in objects_to_build
.
sourcepub fn depends_on_into(
&self,
collection_id: GlobalId,
out: &mut BTreeSet<GlobalId>
)
pub fn depends_on_into(
&self,
collection_id: GlobalId,
out: &mut BTreeSet<GlobalId>
)
Like depends_on
, but appends to an existing BTreeSet
.
sourceimpl<P: PartialEq, S: PartialEq, T: PartialOrder> DataflowDescription<P, S, T>
impl<P: PartialEq, S: PartialEq, T: PartialOrder> DataflowDescription<P, S, T>
sourcepub fn compatible_with(&self, other: &Self) -> bool
pub fn compatible_with(&self, other: &Self) -> bool
Determine if a dataflow description is compatible with this dataflow description.
Compatible dataflows have equal exports, imports, and objects to build. The as_of
of
the receiver has to be less equal the other
as_of
.
Trait Implementations
sourceimpl Arbitrary for DataflowDescription<Plan, CollectionMetadata, Timestamp>
impl Arbitrary for DataflowDescription<Plan, CollectionMetadata, Timestamp>
type Parameters = ()
type Parameters = ()
arbitrary_with
accepts for configuration
of the generated Strategy
. Parameters must implement Default
. Read moresourcefn arbitrary_with(_: Self::Parameters) -> Self::Strategy
fn arbitrary_with(_: Self::Parameters) -> Self::Strategy
sourceimpl<P: Clone, S: Clone, T: Clone> Clone for DataflowDescription<P, S, T>
impl<P: Clone, S: Clone, T: Clone> Clone for DataflowDescription<P, S, T>
sourcefn clone(&self) -> DataflowDescription<P, S, T>
fn clone(&self) -> DataflowDescription<P, S, T>
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moresourceimpl<'de, P, S, T> Deserialize<'de> for DataflowDescription<P, S, T>where
P: Deserialize<'de>,
S: Deserialize<'de>,
T: Deserialize<'de>,
impl<'de, P, S, T> Deserialize<'de> for DataflowDescription<P, S, T>where
P: Deserialize<'de>,
S: Deserialize<'de>,
T: Deserialize<'de>,
sourcefn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
sourceimpl<P: PartialEq, S: PartialEq, T: PartialEq> PartialEq<DataflowDescription<P, S, T>> for DataflowDescription<P, S, T>
impl<P: PartialEq, S: PartialEq, T: PartialEq> PartialEq<DataflowDescription<P, S, T>> for DataflowDescription<P, S, T>
sourcefn eq(&self, other: &DataflowDescription<P, S, T>) -> bool
fn eq(&self, other: &DataflowDescription<P, S, T>) -> bool
sourceimpl RustType<ProtoDataflowDescription> for DataflowDescription<Plan, CollectionMetadata>
impl RustType<ProtoDataflowDescription> for DataflowDescription<Plan, CollectionMetadata>
sourcefn into_proto(&self) -> ProtoDataflowDescription
fn into_proto(&self) -> ProtoDataflowDescription
Self
into a Proto
value.sourcefn from_proto(
proto: ProtoDataflowDescription
) -> Result<Self, TryFromProtoError>
fn from_proto(
proto: ProtoDataflowDescription
) -> Result<Self, TryFromProtoError>
sourceimpl<P, S, T> Serialize for DataflowDescription<P, S, T>where
P: Serialize,
S: Serialize,
T: Serialize,
impl<P, S, T> Serialize for DataflowDescription<P, S, T>where
P: Serialize,
S: Serialize,
T: Serialize,
impl<P: Eq, S: Eq, T: Eq> Eq for DataflowDescription<P, S, T>
impl<P, S, T> StructuralEq for DataflowDescription<P, S, T>
impl<P, S, T> StructuralPartialEq for DataflowDescription<P, S, T>
Auto Trait Implementations
impl<P, S, T> RefUnwindSafe for DataflowDescription<P, S, T>where
P: RefUnwindSafe,
S: RefUnwindSafe,
T: RefUnwindSafe,
impl<P, S, T> Send for DataflowDescription<P, S, T>where
P: Send,
S: Send,
T: Send,
impl<P, S, T> Sync for DataflowDescription<P, S, T>where
P: Sync,
S: Sync,
T: Sync,
impl<P, S, T> Unpin for DataflowDescription<P, S, T>where
P: Unpin,
T: Unpin,
impl<P, S, T> UnwindSafe for DataflowDescription<P, S, T>where
P: UnwindSafe,
S: RefUnwindSafe,
T: UnwindSafe + RefUnwindSafe,
Blanket Implementations
sourceimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
sourceimpl<Q, K> Equivalent<K> for Qwhere
Q: Eq + ?Sized,
K: Borrow<Q> + ?Sized,
impl<Q, K> Equivalent<K> for Qwhere
Q: Eq + ?Sized,
K: Borrow<Q> + ?Sized,
sourcefn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
key
and return true
if they are equal.sourceimpl<T> FutureExt for T
impl<T> FutureExt for T
sourcefn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
sourcefn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
sourceimpl<T> Instrument for T
impl<T> Instrument for T
sourcefn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
sourcefn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
sourceimpl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
sourcefn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T
in a tonic::Request
sourceimpl<T> Pointable for T
impl<T> Pointable for T
sourceimpl<T> ProgressEventTimestamp for Twhere
T: Data + Debug + Any,
impl<T> ProgressEventTimestamp for Twhere
T: Data + Debug + Any,
sourceimpl<P, R> ProtoType<R> for Pwhere
R: RustType<P>,
impl<P, R> ProtoType<R> for Pwhere
R: RustType<P>,
sourcefn into_rust(self) -> Result<R, TryFromProtoError>
fn into_rust(self) -> Result<R, TryFromProtoError>
RustType::from_proto
.sourcefn from_rust(rust: &R) -> P
fn from_rust(rust: &R) -> P
RustType::into_proto
.