Module kafka 

Structs

GlueConsumerContext 🔒

An implementation of ConsumerContext that forwards statistics to the worker

HealthStatus 🔒

KafkaResumeUpperProcessor

Processes resume_uppers stream updates, committing them upstream and reporting each export’s offset_committed.

KafkaSourceReader

Contains all information necessary to ingest data from Kafka

PartitionCapability 🔒

PartitionConsumer 🔒

Wrapper around a partition containing the underlying consumer

ReaderConsumerSetup 🔒

Creates a reader worker’s consumer, possibly long after the operator started, and reports failures to the operator’s outputs.

SourceOutputInfo 🔒

Enums

KafkaHeaderParseError

MetadataUpdate 🔒

An update produced by the metadata fetcher.

Functions

construct_source_message 🔒

fetch_partition_info 🔒

Fetches the list of partitions and their corresponding high watermark.

render_metadata_fetcher 🔒

Render the metadata fetcher of a Kafka source.

render_reader 🔒

Render the reader of a Kafka source.

responsible_for_pid 🔒

Computes whether this worker is responsible for consuming a partition. It assigns partitions to workers in a round-robin fashion, starting at an arbitrary worker based on the hash of the source id.

spawn_metadata_thread 🔒

Type Aliases

DataOutput 🔒

HealthOutput 🔒

PartitionWatermark 🔒

The high/low watermark offsets of a Kafka partition.

ReaderConsumer 🔒