Module mz_kafka_util::client

Expand description

Helpers for working with Kafka’s client API.

Structs§

BrokerAddr
The address of a Kafka broker.
BrokerRewrite
Rewrites a broker address.
MzClientContext
A ClientContext implementation that uses tracing instead of log macros.
TimeoutConfig
Configurable timeouts for Kafka connections.
TunnelingClientContext
A client context that supports rewriting broker addresses.

BrokerRewriteHandle 🔒
GetPartitionsError
The error returned by get_partitions.
MzKafkaError
A structured error type for errors reported by librdkafka through its logs.
TunnelConfig
Tunneling clients used for re-writing ports / hosts

DEFAULT_FETCH_METADATA_TIMEOUT
A reasonable default timeout when fetching metadata or partitions.
DEFAULT_KEEPALIVE
Default to true as they have no downsides https://github.com/confluentinc/librdkafka/issues/283.
DEFAULT_METADATA_FETCH_INTERVAL
The interval we will fetch metadata from, unless overridden by the source.
DEFAULT_PROGRESS_RECORD_FETCH_TIMEOUT
The timeout for reading records from the progress topic. Set to something slightly longer than the idle transaction timeout (60s) to wait out any stuck producers.
DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT
The rdkafka default.
DEFAULT_SOCKET_TIMEOUT
The rdkafka default.
DEFAULT_TOPIC_METADATA_REFRESH_INTERVAL
A reasonable default timeout when refreshing topic metadata. This is configured at a source level.
DEFAULT_TRANSACTION_TIMEOUT
Increased from the rdkafka default

create_new_client_config
Build a new rdkafka ClientConfig with its log_level set correctly based on the passed through tracing::Level. This level should be determined for target: "librdkafka".
create_new_client_config_simple
A simpler version of create_new_client_config that defaults the log_level to INFO and should only be used in tests.
get_partitions
Retrieve number of partitions for a given topic using the given client