Struct mz_rocksdb::config::RocksDBTuningParameters

source ·
pub struct RocksDBTuningParameters {
Show 16 fields pub compaction_style: CompactionStyle, pub optimize_compaction_memtable_budget: usize, pub level_compaction_dynamic_level_bytes: bool, pub universal_compaction_target_ratio: i32, pub parallelism: Option<i32>, pub compression_type: CompressionType, pub bottommost_compression_type: CompressionType, pub batch_size: usize, pub retry_max_duration: Duration, pub stats_log_interval_seconds: u32, pub stats_persist_interval_seconds: u32, pub point_lookup_block_cache_size_mb: Option<u32>, pub shrink_buffers_by_ratio: usize, pub write_buffer_manager_memory_bytes: Option<usize>, pub write_buffer_manager_memory_fraction: Option<f64>, pub write_buffer_manager_allow_stall: bool,
}
Expand description

A set of parameters to tune RocksDB. This struct is plain-old-data, and is used to update RocksDBConfig, which contains some dynamic value for some parameters.

Fields§

§compaction_style: CompactionStyle

RocksDB has 2 primary styles of compaction:

  • The default, usually referred to as “level” compaction
  • “universal”

Universal is simpler and for some workloads could be better. Also, you can directly configure its space-amplification ratio (using universal_compaction_target_ratio). However, its unclear if the UPSERT workload is a good workload for universal compaction, and its also might be the case that universal compaction uses significantly more space temporarily while performing compaction.

For these reasons, the default is CompactionStyle::Level.

§optimize_compaction_memtable_budget: usize

The RocksDB api offers a single configuration method that sets some reasonable defaults for heavy-write workloads, either https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.optimize_level_style_compaction or https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.optimize_universal_style_compaction depending on compaction_style. We ALSO enable this configuration, which is tuned by the size of the memtable (basically the in-memory buffer used to avoid IO). The default here is ~512MB, which is the default from here: https://github.com/facebook/rocksdb/blob/main/include/rocksdb/options.h#L102, and about twice the global RocksDB default.

§level_compaction_dynamic_level_bytes: bool

This option, when enabled, dynamically tunes the size of the various LSM levels to put a bound on space-amplification. With the default level-ratio of 10, this means space-amplification is O(1.11 * the size of data). Note this is big-O notation, and the actual amplification factor depends on the workload.

See https://www.eecg.toronto.edu/~stumm/Papers/Dong-CIDR-16.pdf for more details.

This option defaults to true, as its basically free saved-space, and only applies to CompactionStyle::Level.

§universal_compaction_target_ratio: i32

The additional space-amplification used with universal compaction. Only applies to CompactionStyle::Universal.

See compaction_style for more information.

§parallelism: Option<i32>

By default, RocksDB uses only 1 thread to perform compaction and other background tasks.

The default here is the number of cores, as mentioned by https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.increase_parallelism.

Note that this option is shared across all RocksDB instances that share a rocksdb::Env.

§compression_type: CompressionType

The most important way to reduce space amplification in RocksDB is compression.

In RocksDB, data on disk is stored in an LSM tree. Because the higher layers (which are smaller) will need to be read during reads that aren’t cached, we want a relatively lightweight compression scheme, choosing Lz4 as the default, which is considered almost always better than Snappy.

The meat of the data is stored in the largest, bottom layer, which can be configured (using bottommost_compression_type) to use a more expensive compression scheme to save more space. The default is Zstd, which many think has the best compression ratio. Note that tuning the bottommost layer separately only makes sense when you have free cpu, which we have in the case of the UPSERT usecase.

§bottommost_compression_type: CompressionType

See compression_type for more information.

§batch_size: usize

The size of the multi_get and multi_put batches sent to RocksDB. The default is 1024.

§retry_max_duration: Duration

The maximum duration for the retries when performing rocksdb actions in case of retry-able errors.

§stats_log_interval_seconds: u32

The interval to dump stats in LOG.

§stats_persist_interval_seconds: u32

The interval to persist stats into rocksdb.

§point_lookup_block_cache_size_mb: Option<u32>

The optional block cache size in MiB for optimizing rocksdb for point lookups. If not provided there will be no optimization. https://github.com/facebook/rocksdb/blob/main/include/rocksdb/options.h#L82-L85

§shrink_buffers_by_ratio: usize

The number of times by which unused buffers will be reduced. For example, if the number is 2, the buffers will be reduced to being twice as small, i.e. halved. Shrinking will be disabled if value is 0;

§write_buffer_manager_memory_bytes: Option<usize>

Optional write buffer manager bytes. This needs to be set to enable write buffer manager across all rocksdb instances

§write_buffer_manager_memory_fraction: Option<f64>

Optional write buffer manager memory limit as a percentage of cluster limit

§write_buffer_manager_allow_stall: bool

Config to enable stalls with write buffer manager

Implementations§

source§

impl RocksDBTuningParameters

source

pub fn from_parameters( compaction_style: CompactionStyle, optimize_compaction_memtable_budget: usize, level_compaction_dynamic_level_bytes: bool, universal_compaction_target_ratio: i32, parallelism: Option<i32>, compression_type: CompressionType, bottommost_compression_type: CompressionType, batch_size: usize, retry_max_duration: Duration, stats_log_interval_seconds: u32, stats_persist_interval_seconds: u32, point_lookup_block_cache_size_mb: Option<u32>, shrink_buffers_by_ratio: usize, write_buffer_manager_memory_bytes: Option<usize>, write_buffer_manager_memory_fraction: Option<f64>, write_buffer_manager_allow_stall: bool, ) -> Result<RocksDBTuningParameters, Error>

Build a RocksDBTuningParameters from strings and values from LD parameters.

Trait Implementations§

source§

impl Arbitrary for RocksDBTuningParameters

§

type Parameters = (<CompactionStyle as Arbitrary>::Parameters, <usize as Arbitrary>::Parameters, <bool as Arbitrary>::Parameters, <i32 as Arbitrary>::Parameters, <Option<i32> as Arbitrary>::Parameters, <CompressionType as Arbitrary>::Parameters, <CompressionType as Arbitrary>::Parameters, <usize as Arbitrary>::Parameters, <Duration as Arbitrary>::Parameters, (<u32 as Arbitrary>::Parameters, <u32 as Arbitrary>::Parameters, <Option<u32> as Arbitrary>::Parameters, <usize as Arbitrary>::Parameters, <Option<usize> as Arbitrary>::Parameters, <Option<f64> as Arbitrary>::Parameters, <bool as Arbitrary>::Parameters))

The type of parameters that arbitrary_with accepts for configuration of the generated Strategy. Parameters must implement Default.
§

type Strategy = Map<(<CompactionStyle as Arbitrary>::Strategy, <usize as Arbitrary>::Strategy, <bool as Arbitrary>::Strategy, <i32 as Arbitrary>::Strategy, <Option<i32> as Arbitrary>::Strategy, <CompressionType as Arbitrary>::Strategy, <CompressionType as Arbitrary>::Strategy, <usize as Arbitrary>::Strategy, <Duration as Arbitrary>::Strategy, (<u32 as Arbitrary>::Strategy, <u32 as Arbitrary>::Strategy, <Option<u32> as Arbitrary>::Strategy, <usize as Arbitrary>::Strategy, <Option<usize> as Arbitrary>::Strategy, <Option<f64> as Arbitrary>::Strategy, <bool as Arbitrary>::Strategy)), fn(_: (CompactionStyle, usize, bool, i32, Option<i32>, CompressionType, CompressionType, usize, Duration, (u32, u32, Option<u32>, usize, Option<usize>, Option<f64>, bool))) -> RocksDBTuningParameters>

The type of Strategy used to generate values of type Self.
source§

fn arbitrary_with( _top: <RocksDBTuningParameters as Arbitrary>::Parameters, ) -> <RocksDBTuningParameters as Arbitrary>::Strategy

Generates a Strategy for producing arbitrary values of type the implementing type (Self). The strategy is passed the arguments given in args. Read more
source§

fn arbitrary() -> Self::Strategy

Generates a Strategy for producing arbitrary values of type the implementing type (Self). Read more
source§

impl Clone for RocksDBTuningParameters

source§

fn clone(&self) -> RocksDBTuningParameters

Returns a copy of the value. Read more
1.0.0 · source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
source§

impl Debug for RocksDBTuningParameters

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more
source§

impl Default for RocksDBTuningParameters

source§

fn default() -> RocksDBTuningParameters

Returns the “default value” for a type. Read more
source§

impl<'de> Deserialize<'de> for RocksDBTuningParameters

source§

fn deserialize<__D>( __deserializer: __D, ) -> Result<RocksDBTuningParameters, <__D as Deserializer<'de>>::Error>
where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer. Read more
source§

impl PartialEq for RocksDBTuningParameters

source§

fn eq(&self, other: &RocksDBTuningParameters) -> bool

Tests for self and other values to be equal, and is used by ==.
1.0.0 · source§

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.
source§

impl RustType<ProtoRocksDbTuningParameters> for RocksDBTuningParameters

source§

fn into_proto(&self) -> ProtoRocksDbTuningParameters

Convert a Self into a Proto value.
source§

fn from_proto( proto: ProtoRocksDbTuningParameters, ) -> Result<RocksDBTuningParameters, TryFromProtoError>

Consume and convert a Proto back into a Self value. Read more
source§

fn into_proto_owned(self) -> Proto

A zero clone version of Self::into_proto that types can optionally implement, otherwise, the default implementation delegates to Self::into_proto.
source§

impl Serialize for RocksDBTuningParameters

source§

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>
where __S: Serializer,

Serialize this value into the given Serde serializer. Read more
source§

impl StructuralPartialEq for RocksDBTuningParameters

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T, U> CastInto<U> for T
where U: CastFrom<T>,

source§

fn cast_into(self) -> U

Performs the cast.
source§

impl<T> CloneToUninit for T
where T: Clone,

source§

default unsafe fn clone_to_uninit(&self, dst: *mut T)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dst. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T> FromRef<T> for T
where T: Clone,

source§

fn from_ref(input: &T) -> T

Converts to this type from a reference to the input type.
source§

impl<T> FutureExt for T

source§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
source§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
source§

impl<T> Instrument for T

source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> IntoRequest<T> for T

source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
source§

impl<T> Pointable for T

source§

const ALIGN: usize = _

The alignment of pointer.
§

type Init = T

The type for initializers.
source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
source§

impl<P, R> ProtoType<R> for P
where R: RustType<P>,

source§

impl<T> Same for T

§

type Output = T

Should always be Self
source§

impl<T> ToOwned for T
where T: Clone,

§

type Owned = T

The resulting type after obtaining ownership.
source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

source§

fn vzip(self) -> V

source§

impl<T> WithSubscriber for T

source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

impl<T> Allocation for T
where T: RefUnwindSafe + Send + Sync,

source§

impl<T> DeserializeOwned for T
where T: for<'de> Deserialize<'de>,