Module mz_storage::source::kafka

Structs

• GlueConsumerContext 🔒
  An implementation of ConsumerContext that forwards statistics to the worker
• HealthStatus 🔒
• KafkaResumeUpperProcessor
  Processes resume_uppers stream updates, committing them upstream and storing them in the progress_statistics to be emitted later.
• KafkaSourceReader
  Contains all information necessary to ingest data from Kafka
• MetadataUpdate 🔒
  An update produced by the metadata fetcher.
• PartialProgressStatistics 🔒
  A partially-filled version of ProgressStatisticsUpdate. This allows us to only emit updates when offset_known is updated by the metadata thread.
• PartitionCapability 🔒
• PartitionConsumer 🔒
  Wrapper around a partition containing the underlying consumer
• SourceOutputInfo 🔒

Enums

• KafkaHeaderParseError

Functions

• construct_source_message 🔒
• fetch_partition_info 🔒
  Fetches the list of partitions and their corresponding high watermark.
• render_metadata_fetcher 🔒
  Render the metadata fetcher of a Kafka source.
• render_reader 🔒
  Render the reader of a Kafka source.
• responsible_for_pid 🔒
  Computes whether this worker is responsible for consuming a partition. It assigns partitions to workers in a round-robin fashion, starting at an arbitrary worker based on the hash of the source id.
• spawn_metadata_thread 🔒

Type Aliases

• HighWatermark 🔒
  The high watermark offsets of a Kafka partition.