Module client

Expand description

Helpers for working with Kafka’s client API.

Structs§

BrokerAddr: The address of a Kafka broker.
BrokerRewrite: Rewrites a broker address.
MzClientContext: A ClientContext implementation that uses tracing instead of log macros.
TimeoutConfig: Configurable timeouts for Kafka connections.
TunnelingClientContext: A client context that supports rewriting broker addresses.

BrokerRewriteHandle 🔒
GetPartitionsError: The error returned by get_partitions.
MzKafkaError: A structured error type for errors reported by librdkafka through its logs.
TunnelConfig: Tunneling clients used for re-writing ports / hosts

DEFAULT_FETCH_METADATA_TIMEOUT: A reasonable default timeout when fetching metadata or partitions.
DEFAULT_KEEPALIVE: Default to true as they have no downsides https://github.com/confluentinc/librdkafka/issues/283.
DEFAULT_PROGRESS_RECORD_FETCH_TIMEOUT: The timeout for reading records from the progress topic. Set to something slightly longer than the idle transaction timeout (60s) to wait out any stuck producers.
DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT: The rdkafka default.
DEFAULT_SOCKET_TIMEOUT: The rdkafka default.
DEFAULT_TOPIC_METADATA_REFRESH_INTERVAL: A reasonable default timeout when refreshing topic metadata. This is configured at a source level.
DEFAULT_TRANSACTION_TIMEOUT: Increased from the rdkafka default

create_new_client_config: Build a new rdkafka ClientConfig with its log_level set correctly based on the passed through tracing::Level. This level should be determined for target: "librdkafka".
create_new_client_config_simple: A simpler version of create_new_client_config that defaults the log_level to INFO and should only be used in tests.
get_partitions: Retrieve number of partitions for a given topic using the given client