Module mz_storage::source::mysql

Code to render the ingestion dataflow of a MySqlSourceConnection.

This dataflow is split into Snapshot and Replication operators.

Snapshot

The snapshot operator is responsible for taking a consistent snapshot of the tables involved in the ingestion from the MySQL server. Each table that is being ingested is snapshot is assigned a specific worker, which performs a SELECT * FROM table and emits updates for all the rows in the given table.

For all tables that are snapshotted the snapshot operator also emits a rewind request to the replication operator containing the GTID-set based frontier which will be used to ensure that the requested portion of the replication stream is subtracted from the snapshot.

See the snapshot module for more information.

Replication

The replication operator is responsible for ingesting the MySQL replication stream which must happen from a single worker.

See the replication module for more information.

Error handling

There are two kinds of errors that can happen during ingestion that are represented as two separate error types:

DefiniteErrors are errors that happen during processing of a specific collection record. These are the only errors that can ever end up in the error collection of a subsource.

TransientErrors are any errors that can happen for reasons that are unrelated to the data itself. This could be authentication failures, connection failures, etc. The only operators that can emit such errors are the MySqlReplicationReader and the MySqlSnapshotReader operators, which are the ones that talk to the external world. Both of these operators are built with the AsyncOperatorBuilder::build_fallible method which allows transient errors to be propagated upwards with the standard ? operator without risking downgrading the capability and producing bogus frontiers.

The error streams from both of those operators are published to the source status and also trigger a restart of the dataflow.

Modules

• replication 🔒
  Renders the replication side of the MySqlSourceConnection ingestion dataflow.
• schemas 🔒
• snapshot 🔒
  Renders the table snapshot side of the MySqlSourceConnection dataflow.
• statistics 🔒
  Renders the statistics collection of the MySqlSourceConnection ingestion dataflow.

Structs

• MySqlTableName 🔒
  A reference to a MySQL table. (schema_name, table_name) NOTE: We do not use mz_sql_parser::ast:UnresolvedItemName because the serialization behavior is not what we need for mysql.
• RewindRequest 🔒
• SourceOutputInfo 🔒

Enums

• DefiniteError
  A definite error that always ends up in the collection of a specific table.
• ReplicationError
• TransientError
  A transient error that never ends up in the collection of a specific table.

Functions

• return_definite_error 🔒
• validate_mysql_repl_settings 🔒

Type Aliases

• StackedAsyncOutputHandle 🔒