Module optimize

Source
Expand description

Optimizer interface to the adapter and coordinator code.

The goal of this crate is to abstract optimizer specifics behind a high-level interface that is ready to be consumed by the coordinator code in a future-proof way (that is, the API is taking the upcoming evolution of these components into account).

The contents of this crate should have minimal dependencies to the rest of the coordinator code so we can pull them out as a separate crate in the future without too much effort.

The main type in this module is a very simple Optimize trait which allows us to adhere to the following principles:

  • Implementors of this trait are structs that encapsulate all context required to optimize a statement of type T end-to-end (for example materialized_view::Optimizer for T = MaterializedView).
  • Each struct implements Optimize once for each optimization stage. The From type represents the input of the stage and Self::To the associated stage output. This allows to have more than one entrypoints to a pipeline.
  • The concrete types used for stage results are opaque structs that are specific to the pipeline of that statement type.
    • We use different structs even if two statement types might have structurally identical intermediate results. This ensures that client code cannot first execute some optimization stages for one type and then some stages for a different type.
    • The only way to construct such a struct is by running the Optimize stage that produces it. This ensures that client code cannot interfere with the pipeline.
    • In general, the internals of these structs can be accessed only behind a shared reference. This ensures that client code can look up information from intermediate stages but cannot modify it.
    • Timestamp selection is modeled as a conversion between structs that are adjacent in the pipeline using a method called resolve.
    • The struct representing the result of the final stage of the optimization pipeline can be destructed to access its internals with a method called unapply.
  • The Send trait bounds on the Self and From types ensure that Optimize instances can be passed to different threads (this is required of off-thread optimization).

For details, see the 20230714_optimizer_interface.md design doc in this repository.

Modulesยง

copy_to
Optimizer implementation for COPY TO statements.
dataflows
Types and methods for building and shipping dataflow descriptions.
index
Optimizer implementation for CREATE INDEX statements.
materialized_view
Optimizer implementation for CREATE MATERIALIZED VIEW statements.
peek
Optimizer implementation for SELECT statements.
subscribe
Optimizer implementation for SUBSCRIBE statements.
view
An Optimizer that

Macrosยง

trace_plan ๐Ÿ”’

Structsยง

OptimizerConfig
Feature flags for the optimizer.

Enumsยง

OptimizeMode
OptimizerError
Error types that can be generated during optimization.

Traitsยง

Optimize
A trait that represents an optimization stage.
OptimizerCatalog

Functionsยง

optimize_mir_constant ๐Ÿ”’
This is just a wrapper around mz_transform::Optimizer::constant_optimizer, running it, and tracing the result plan.
optimize_mir_local ๐Ÿ”’

Type Aliasesยง

LirDataflowDescription ๐Ÿ”’
A type for a DataflowDescription backed by Lir~ plans. Used internally by the optimizer implementations.
MirDataflowDescription ๐Ÿ”’
A type for a DataflowDescription backed by Mir~ plans. Used internally by the optimizer implementations.