Skip to main content

prune_dataflow_source_imports

Function prune_dataflow_source_imports 

Source
fn prune_dataflow_source_imports(dataflow: &mut DataflowDesc)
Expand description

Restricts the sources imported by dataflow to only the ones its exports read.

The counterpart to prune_and_annotate_dataflow_index_imports for source imports. Imports are collected before the global pipeline runs, from the plans as they were written, so a transform can drop the last Get of one, for instance by folding a selection to a constant.

An import that survives that is not free. Every worker builds a persist_source for it and decodes a shard into a stream nobody consumes, and the controller takes a read hold that pins the collection’s since for as long as the dataflow lives. Both are read off the import list directly, so pruning is what reclaims them.

A third consumer, the wall-clock dependence a dataflow reports, is the one whose wrong answer does real damage: it earns a dataflow whose exports can never change again an expiration, which pins their output frontier at the expiration time rather than letting it reach the empty antichain, so nothing downstream learns the collection is final. ComputeController::determine_time_dependence derives that from the read set rather than from the import list, so it does not depend on this pass having run. create_dataflow also reports a list this pass left loose.

The input plans should be normalized with NormalizeLets, for the same reason prune_and_annotate_dataflow_index_imports wants them to be: an unused Let binding can otherwise keep alive a Get that nothing reads.