Module dictionary 

A Row-specialized container using dictionary compression.

The approach is to establish for each column lists of common values, and to use “unoccupied” tags in the row encoding (e.g. where we would indicate types) to replace these common values. This substitution is opt-in, in that we don’t need to do it, and in particular do not do it while we are collecting preliminary information about common values, and then start to use it once we believe we have enough information. Once we have started to use the substitutions we cannot change the meaning of a reserved byte pattern, for the container we are populating.

Each from-scratch container observes STATS_THRESHOLD records before establishing a mapping from spare tags to common values. Containers that are formed from merging other containers use those input containers’ common values to populate a codec and use it immediately.

The dictionary behavior is controlled by the DICTIONARY_COMPRESSION flag, which if disabled prevents the construction of codecs, which when absent simply cause the wrapper to behave as a no-op that fails to use any spare tags for common values. The flag is set once, when a replica is created (from compute’s InstanceConfig::arrangement_dictionary_compression, itself captured from the enable_arrangement_dictionary_compression_alpha dyncfg at that moment), and is not changed for the life of the process; flipping the dyncfg only affects replicas created afterwards. Even with the flag fixed, a single replica can hold a mix of compressed and uncompressed containers — e.g. containers that never observed enough records to install a codec, or that were merged from uncompressed inputs.

Modules

builders

Wrapper types that exist to support the creation of dictionary codecs.

Structs

DatumContainer

DatumSeq

A reference that can be resolved to a sequence of Datums.

Constants

STATS_THRESHOLD 🔒

Number of pushes a from-scratch container observes before it turns its gathered stats into a safe codec.