Struct mz_storage::source::postgres::PostgresTaskInfo

struct PostgresTaskInfo {
    source_id: GlobalId,
    connection_config: Config,
    publication: String,
    slot: String,
    replication_lsn: PgLsn,
    metrics: PgSourceMetrics,
    source_tables: BTreeMap<u32, SourceTable>,
    row_sender: RowSender,
    sender: Sender<(usize, InternalMessage)>,
    resume_lsn: Arc<AtomicU64>,
}

An internal struct held by the spawned tokio task

Fields

source_id: GlobalId

connection_config: Config

publication: String

slot: String

replication_lsn: PgLsn

Our cursor into the WAL

metrics: PgSourceMetrics

source_tables: BTreeMap<u32, SourceTable>

A map of the table oid to its information.

Note that we populate this information with state from the catalog, but only remove it if we encounter errors during execution. This means it is possible for items to be removed during one execution cycle, only to have them re-appear on restart. For instance, if we remove a table because of an issue with its schema, and the operator then “fixes” the issue with the schema, we will not remove the table during the next execution cycle. This isn’t a large concern because the fact that we never retract errors from subsources means that the source never return readable values, though it will start sending data along its Ok stream again.

At the time of writing, the plan is to resolve the above issue when we track each source table’s frontier independently; in that world, we can close the source table’s frontier so we have a durable signal that it should never produce any data.

row_sender: RowSender

sender: Sender<(usize, InternalMessage)>

resume_lsn: Arc<AtomicU64>