Struct parquet::arrow::arrow_reader::RowFilter
source · pub struct RowFilter { /* private fields */ }
Expand description
A RowFilter
allows pushing down a filter predicate to skip IO and decode
This consists of a list of ArrowPredicate
where only the rows that satisfy all
of the predicates will be returned. Any RowSelection
will be applied prior
to the first predicate, and each predicate in turn will then be used to compute
a more refined RowSelection
to use when evaluating the subsequent predicates.
Once all predicates have been evaluated, the final RowSelection
is applied
to the top-level ProjectionMask
to produce the final output RecordBatch
.
This design has a couple of implications:
RowFilter
can be used to skip entire pages, and thus IO, in addition to CPU decode overheads- Columns may be decoded multiple times if they appear in multiple
ProjectionMask
- IO will be deferred until needed by a
ProjectionMask
As such there is a trade-off between a single large predicate, or multiple predicates, that will depend on the shape of the data. Whilst multiple smaller predicates may minimise the amount of data scanned/decoded, it may not be faster overall.
For example, if a predicate that needs a single column of data filters out all but
1% of the rows, applying it as one of the early ArrowPredicateFn
will likely significantly
improve performance.
As a counter example, if a predicate needs several columns of data to evaluate but leaves 99% of the rows, it may be better to not filter the data from parquet and apply the filter after the RecordBatch has been fully decoded.
Additionally, even if a predicate eliminates a moderate number of rows, it may still be faster to filter the data after the RecordBatch has been fully decoded, if the eliminated rows are not contiguous.
Implementations§
source§impl RowFilter
impl RowFilter
sourcepub fn new(predicates: Vec<Box<dyn ArrowPredicate>>) -> Self
pub fn new(predicates: Vec<Box<dyn ArrowPredicate>>) -> Self
Create a new RowFilter
from an array of ArrowPredicate