Struct mz_rocksdb::config::RocksDBTuningParameters

pub struct RocksDBTuningParameters {

    pub compaction_style: CompactionStyle,
    pub optimize_compaction_memtable_budget: usize,
    pub level_compaction_dynamic_level_bytes: bool,
    pub universal_compaction_target_ratio: i32,
    pub parallelism: Option<i32>,
    pub compression_type: CompressionType,
    pub bottommost_compression_type: CompressionType,
    pub batch_size: usize,
    pub retry_max_duration: Duration,
    pub stats_log_interval_seconds: u32,
    pub stats_persist_interval_seconds: u32,
    pub point_lookup_block_cache_size_mb: Option<u32>,
    pub shrink_buffers_by_ratio: usize,
    pub write_buffer_manager_memory_bytes: Option<usize>,
    pub write_buffer_manager_memory_fraction: Option<f64>,
    pub write_buffer_manager_allow_stall: bool,
}

A set of parameters to tune RocksDB. This struct is plain-old-data, and is used to update RocksDBConfig, which contains some dynamic value for some parameters.

Fields

compaction_style: CompactionStyle

RocksDB has 2 primary styles of compaction:

• The default, usually referred to as “level” compaction
• “universal”

Universal is simpler and for some workloads could be better. Also, you can directly configure its space-amplification ratio (using universal_compaction_target_ratio). However, its unclear if the UPSERT workload is a good workload for universal compaction, and its also might be the case that universal compaction uses significantly more space temporarily while performing compaction.

For these reasons, the default is CompactionStyle::Level.

optimize_compaction_memtable_budget: usize

The RocksDB api offers a single configuration method that sets some reasonable defaults for heavy-write workloads, either https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.optimize_level_style_compaction or https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.optimize_universal_style_compaction depending on compaction_style. We ALSO enable this configuration, which is tuned by the size of the memtable (basically the in-memory buffer used to avoid IO). The default here is ~512MB, which is the default from here: https://github.com/facebook/rocksdb/blob/main/include/rocksdb/options.h#L102, and about twice the global RocksDB default.

level_compaction_dynamic_level_bytes: bool

This option, when enabled, dynamically tunes the size of the various LSM levels to put a bound on space-amplification. With the default level-ratio of 10, this means space-amplification is O(1.11 * the size of data). Note this is big-O notation, and the actual amplification factor depends on the workload.

See https://www.eecg.toronto.edu/~stumm/Papers/Dong-CIDR-16.pdf for more details.

This option defaults to true, as its basically free saved-space, and only applies to CompactionStyle::Level.

universal_compaction_target_ratio: i32

The additional space-amplification used with universal compaction. Only applies to CompactionStyle::Universal.

See compaction_style for more information.

parallelism: Option<i32>

By default, RocksDB uses only 1 thread to perform compaction and other background tasks.

The default here is the number of cores, as mentioned by https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.increase_parallelism.

Note that this option is shared across all RocksDB instances that share a rocksdb::Env.

compression_type: CompressionType

The most important way to reduce space amplification in RocksDB is compression.

In RocksDB, data on disk is stored in an LSM tree. Because the higher layers (which are smaller) will need to be read during reads that aren’t cached, we want a relatively lightweight compression scheme, choosing Lz4 as the default, which is considered almost always better than Snappy.

The meat of the data is stored in the largest, bottom layer, which can be configured (using bottommost_compression_type) to use a more expensive compression scheme to save more space. The default is Zstd, which many think has the best compression ratio. Note that tuning the bottommost layer separately only makes sense when you have free cpu, which we have in the case of the UPSERT usecase.

bottommost_compression_type: CompressionType

See compression_type for more information.

batch_size: usize

The size of the multi_get and multi_put batches sent to RocksDB. The default is 1024.

retry_max_duration: Duration

The maximum duration for the retries when performing rocksdb actions in case of retry-able errors.

stats_log_interval_seconds: u32

The interval to dump stats in LOG.

stats_persist_interval_seconds: u32

The interval to persist stats into rocksdb.

point_lookup_block_cache_size_mb: Option<u32>

The optional block cache size in MiB for optimizing rocksdb for point lookups. If not provided there will be no optimization. https://github.com/facebook/rocksdb/blob/main/include/rocksdb/options.h#L82-L85

shrink_buffers_by_ratio: usize

The number of times by which unused buffers will be reduced. For example, if the number is 2, the buffers will be reduced to being twice as small, i.e. halved. Shrinking will be disabled if value is 0;

write_buffer_manager_memory_bytes: Option<usize>

Optional write buffer manager bytes. This needs to be set to enable write buffer manager across all rocksdb instances

write_buffer_manager_memory_fraction: Option<f64>

Optional write buffer manager memory limit as a percentage of cluster limit

write_buffer_manager_allow_stall: bool

Config to enable stalls with write buffer manager

Implementations

impl RocksDBTuningParameters

pub fn from_parameters( compaction_style: CompactionStyle, optimize_compaction_memtable_budget: usize, level_compaction_dynamic_level_bytes: bool, universal_compaction_target_ratio: i32, parallelism: Option<i32>, compression_type: CompressionType, bottommost_compression_type: CompressionType, batch_size: usize, retry_max_duration: Duration, stats_log_interval_seconds: u32, stats_persist_interval_seconds: u32, point_lookup_block_cache_size_mb: Option<u32>, shrink_buffers_by_ratio: usize, write_buffer_manager_memory_bytes: Option<usize>, write_buffer_manager_memory_fraction: Option<f64>, write_buffer_manager_allow_stall: bool, ) -> Result<RocksDBTuningParameters, Error>

Build a RocksDBTuningParameters from strings and values from LD parameters.

Trait Implementations

impl Arbitrary for RocksDBTuningParameters

type Parameters = (<CompactionStyle as Arbitrary>::Parameters, <usize as Arbitrary>::Parameters, <bool as Arbitrary>::Parameters, <i32 as Arbitrary>::Parameters, <Option<i32> as Arbitrary>::Parameters, <CompressionType as Arbitrary>::Parameters, <CompressionType as Arbitrary>::Parameters, <usize as Arbitrary>::Parameters, <Duration as Arbitrary>::Parameters, (<u32 as Arbitrary>::Parameters, <u32 as Arbitrary>::Parameters, <Option<u32> as Arbitrary>::Parameters, <usize as Arbitrary>::Parameters, <Option<usize> as Arbitrary>::Parameters, <Option<f64> as Arbitrary>::Parameters, <bool as Arbitrary>::Parameters))

The type of parameters that arbitrary_with accepts for configuration of the generated Strategy. Parameters must implement Default.

type Strategy = Map<(<CompactionStyle as Arbitrary>::Strategy, <usize as Arbitrary>::Strategy, <bool as Arbitrary>::Strategy, <i32 as Arbitrary>::Strategy, <Option<i32> as Arbitrary>::Strategy, <CompressionType as Arbitrary>::Strategy, <CompressionType as Arbitrary>::Strategy, <usize as Arbitrary>::Strategy, <Duration as Arbitrary>::Strategy, (<u32 as Arbitrary>::Strategy, <u32 as Arbitrary>::Strategy, <Option<u32> as Arbitrary>::Strategy, <usize as Arbitrary>::Strategy, <Option<usize> as Arbitrary>::Strategy, <Option<f64> as Arbitrary>::Strategy, <bool as Arbitrary>::Strategy)), fn(_: (CompactionStyle, usize, bool, i32, Option<i32>, CompressionType, CompressionType, usize, Duration, (u32, u32, Option<u32>, usize, Option<usize>, Option<f64>, bool))) -> RocksDBTuningParameters>

The type of Strategy used to generate values of type Self.

fn arbitrary_with( _top: <RocksDBTuningParameters as Arbitrary>::Parameters, ) -> <RocksDBTuningParameters as Arbitrary>::Strategy

Generates a Strategy for producing arbitrary values of type the implementing type (Self). The strategy is passed the arguments given in args.

fn arbitrary() -> Self::Strategy

Generates a Strategy for producing arbitrary values of type the implementing type (Self).

impl Clone for RocksDBTuningParameters

fn clone(&self) -> RocksDBTuningParameters

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RocksDBTuningParameters

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for RocksDBTuningParameters

fn default() -> RocksDBTuningParameters

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for RocksDBTuningParameters

fn deserialize<__D>( __deserializer: __D, ) -> Result<RocksDBTuningParameters, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for RocksDBTuningParameters

fn eq(&self, other: &RocksDBTuningParameters) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl RustType<ProtoRocksDbTuningParameters> for RocksDBTuningParameters

fn into_proto(&self) -> ProtoRocksDbTuningParameters

Convert a Self into a Proto value.

fn from_proto( proto: ProtoRocksDbTuningParameters, ) -> Result<RocksDBTuningParameters, TryFromProtoError>

Consume and convert a Proto back into a Self value.

fn into_proto_owned(self) -> Proto

A zero clone version of Self::into_proto that types can optionally implement, otherwise, the default implementation delegates to Self::into_proto.

impl Serialize for RocksDBTuningParameters

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for RocksDBTuningParameters