Struct dataflow_types::types::DataflowDescription

pub struct DataflowDescription<View> {
    pub source_imports: BTreeMap<GlobalId, SourceInstanceDesc>,
    pub index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType)>,
    pub objects_to_build: Vec<BuildDesc<View>>,
    pub index_exports: Vec<(GlobalId, IndexDesc, RelationType)>,
    pub sink_exports: Vec<(GlobalId, SinkDesc)>,
    pub dependent_objects: BTreeMap<GlobalId, Vec<GlobalId>>,
    pub as_of: Option<Antichain<Timestamp>>,
    pub debug_name: String,
}

A description of a dataflow to construct and results to surface.

Fields

source_imports: BTreeMap<GlobalId, SourceInstanceDesc>

Sources instantiations made available to the dataflow.

index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType)>

Indexes made available to the dataflow.

objects_to_build: Vec<BuildDesc<View>>

Views and indexes to be built and stored in the local context. Objects must be built in the specific order, as there may be dependencies of later objects on prior identifiers.

index_exports: Vec<(GlobalId, IndexDesc, RelationType)>

Indexes to be made available to be shared with other dataflows (id of new index, description of index, relationtype of base source/view)

sink_exports: Vec<(GlobalId, SinkDesc)>

sinks to be created (id of new sink, description of sink)

dependent_objects: BTreeMap<GlobalId, Vec<GlobalId>>

Maps views to views + indexes needed to generate that view

as_of: Option<Antichain<Timestamp>>

An optional frontier to which inputs should be advanced.

If this is set, it should override the default setting determined by the upper bound of since frontiers contributing to the dataflow. It is an error for this to be set to a frontier not beyond that default.

debug_name: String

Human readable name

Implementations

impl DataflowDescription<OptimizedMirRelationExpr>

pub fn new(name: String) -> Self

Creates a new dataflow description with a human-readable name.

pub fn import_index(
    &mut self,
    id: GlobalId,
    description: IndexDesc,
    typ: RelationType,
    requesting_view: GlobalId
)

Imports a previously exported index.

This method makes available an index previously exported as id, identified to the query by description (which names the view the index arranges, and the keys by which it is arranged).

The requesting_view argument is currently necessary to correctly track the dependencies of views on indexes.

pub fn import_source(
    &mut self,
    id: GlobalId,
    description: SourceDesc,
    persist: Option<SourcePersistDesc>
)

Imports a source and makes it available as id.

pub fn insert_view(&mut self, id: GlobalId, view: OptimizedMirRelationExpr)

Binds to id the relation expression view.

pub fn export_index(
    &mut self,
    id: GlobalId,
    description: IndexDesc,
    on_type: RelationType
)

Exports as id an index on on_id.

Future uses of import_index in other dataflow descriptions may use id, as long as this dataflow has not been terminated in the meantime.

pub fn export_sink(&mut self, id: GlobalId, description: SinkDesc)

Exports as id a sink described by description.

fn record_depends_on(&mut self, view_id: GlobalId, depended_upon: GlobalId)

Records a dependency of view_id on depended_upon.

pub fn is_imported(&self, id: &GlobalId) -> bool

Returns true iff id is already imported.

pub fn set_as_of(&mut self, as_of: Antichain<Timestamp>)

Assigns the as_of frontier to the supplied argument.

This method allows the dataflow to indicate a frontier up through which all times should be advanced. This can be done for at least two reasons: 1. correctness and 2. performance.

Correctness may require an as_of to ensure that historical detail is consolidated at representative times that do not present specific detail that is not specifically correct. For example, updates may be compacted to times that are no longer the source times, but instead some byproduct of when compaction was executed; we should not present those specific times as meaningfully different from other equivalent times.

Performance may benefit from an aggressive as_of as it reduces the number of distinct moments at which collections vary. Differential dataflow will refresh its outputs at each time its inputs change and to moderate that we can minimize the volume of distinct input times as much as possible.

Generally, one should consider setting as_of at least to the since frontiers of contributing data sources and as aggressively as the computation permits.

pub fn arity_of(&self, id: &GlobalId) -> usize

The number of columns associated with an identifier in the dataflow.

impl<View> DataflowDescription<View>

pub fn get_all_imports(&self) -> HashSet<GlobalId>

Collects all indexes required to construct all exports.

pub fn get_imports(&self, id: &GlobalId) -> HashSet<GlobalId>

Collects all transitively dependent identifiers that do not have their own dependencies.

Trait Implementations

impl<View: Clone> Clone for DataflowDescription<View>

fn clone(&self) -> DataflowDescription<View>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<View: Debug> Debug for DataflowDescription<View>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<View: Default> Default for DataflowDescription<View>

fn default() -> DataflowDescription<View>

Returns the “default value” for a type.

impl<'de, View> Deserialize<'de> for DataflowDescription<View> where
    View: Deserialize<'de>, 

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl<View> Serialize for DataflowDescription<View> where
    View: Serialize, 

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.