Module kafka

Structs

GlueConsumerContext 🔒

An implementation of ConsumerContext that forwards statistics to the worker

HealthStatus 🔒

KafkaResumeUpperProcessor

Processes resume_uppers stream updates, committing them upstream and storing them in the progress_statistics to be emitted later.

KafkaSourceReader

Contains all information necessary to ingest data from Kafka

PartialProgressStatistics 🔒

A partially-filled version of ProgressStatisticsUpdate. This allows us to only emit updates when offset_known is updated by the metadata thread.

PartitionCapability 🔒

PartitionConsumer 🔒

Wrapper around a partition containing the underlying consumer

SourceOutputInfo 🔒

Enums

KafkaHeaderParseError

MetadataUpdate 🔒

An update produced by the metadata fetcher.

Functions

construct_source_message 🔒

fetch_partition_info 🔒

Fetches the list of partitions and their corresponding high watermark.

render_metadata_fetcher 🔒

Render the metadata fetcher of a Kafka source.

render_reader 🔒

Render the reader of a Kafka source.

responsible_for_pid 🔒

Computes whether this worker is responsible for consuming a partition. It assigns partitions to workers in a round-robin fashion, starting at an arbitrary worker based on the hash of the source id.

spawn_metadata_thread 🔒

Type Aliases

HighWatermark 🔒

The high watermark offsets of a Kafka partition.