Module mz_adapter::optimize

source ·
Expand description

Optimizer interface to the adapter and coordinator code.

The goal of this crate is to abstract optimizer specifics behind a high-level interface that is ready to be consumed by the coordinator code in a future-proof way (that is, the API is taking the upcoming evolution of these components into account).

The contents of this crate should have minimal dependencies to the rest of the coordinator code so we can pull them out as a separate crate in the future without too much effort.

The main type in this module is a very simple Optimize trait which allows us to adhere to the following principles:

  • Implementors of this trait are structs that encapsulate all context required to optimize a statement of type T end-to-end (for example materialized_view::Optimizer for T = MaterializedView).
  • Each struct implements Optimize once for each optimization stage. The From type represents the input of the stage and Self::To the associated stage output. This allows to have more than one entrypoints to a pipeline.
  • The concrete types used for stage results are opaque structs that are specific to the pipeline of that statement type.
    • We use different structs even if two statement types might have structurally identical intermediate results. This ensures that client code cannot first execute some optimization stages for one type and then some stages for a different type.
    • The only way to construct such a struct is by running the Optimize stage that produces it. This ensures that client code cannot interfere with the pipeline.
    • In general, the internals of these structs can be accessed only behind a shared reference. This ensures that client code can look up information from intermediate stages but cannot modify it.
    • Timestamp selection is modeled as a conversion between structs that are adjacent in the pipeline using a method called resolve.
    • The struct representing the result of the final stage of the optimization pipeline can be destructed to access its internals with a method called unapply.
  • The Send trait bounds on the Self and From types ensure that Optimize instances can be passed to different threads (this is required of off-thread optimization).

For details, see the 20230714_optimizer_interface.md design doc in this repository.

Modules

  • copy_to
    Optimizer implementation for COPY TO statements.
  • dataflows
    Types and methods for building and shipping dataflow descriptions.
  • index
    Optimizer implementation for CREATE INDEX statements.
  • materialized_view
    Optimizer implementation for CREATE MATERIALIZED VIEW statements.
  • peek
    Optimizer implementation for SELECT statements.
  • subscribe
    Optimizer implementation for SUBSCRIBE statements.
  • view
    Optimizer implementation for CREATE VIEW statements.

Macros

Structs

Enums

Traits

  • Optimize
    A trait that represents an optimization stage.
  • OverrideFrom
    A trait used to implement layered config construction.

Functions

Type Aliases