Skip to main content

Module rfc_6678_simulate_layer

opendal_core::docs::rfcs

Module rfc_6678_simulate_layer 

Source
Expand description

Simulate Layer

§Summary

Introduce a public SimulateLayer to replace the internal CompleteLayer, giving users explicit control over capability simulation while maintaining backward compatibility through a phased migration strategy.

§Motivation

Currently, OpenDAL automatically applies CompleteLayer internally to simulate missing capabilities for all backends. While this provides a seamless “batteries-included” experience, it has several limitations:

  1. Lack of user control: Users cannot opt-out of simulations they don’t need, even in performance-sensitive scenarios where native-only operations are preferred.

  2. Hidden complexity: The CompleteLayer has grown to 385 lines handling multiple concerns (list recursive, stat dir, create_dir, reader/writer wrappers), making it difficult to maintain and understand.

  3. All-or-nothing approach: Users either get all simulations or none. There’s no way to selectively enable only the simulations they need.

  4. Binding limitations: While Python/Java/Ruby bindings automatically benefit from simulations, they also cannot disable them, potentially causing unexpected behavior when users expect native backend semantics.

  5. Missing features: Some useful simulations (like start_after for fs backend, reported in #6676) are not implemented because adding more logic to CompleteLayer would make it even more complex.

This proposal aims to:

  • Make capability simulation explicit and user-controllable
  • Reduce complexity by separating concerns
  • Provide a clear migration path that maintains backward compatibility
  • Enable fine-grained control over which simulations to apply

§Guide-level explanation

§For Rust users

After this change, capability simulation becomes explicit through SimulateLayer:

use opendal::layers::SimulateLayer;
use opendal::services::Fs;
use opendal::Operator;

// Default: all simulations enabled
let op = Operator::new(Fs::default())?
    .layer(SimulateLayer::default())
    .finish();

// Selective simulation with method chaining
let op = Operator::new(Fs::default())?
    .layer(
        SimulateLayer::default()
            .with_list_recursive(true)      // Enable recursive listing simulation
            .with_list_start_after(true)    // Enable start_after simulation
            .with_stat_dir(false)           // Disable stat dir simulation
            .with_create_dir(false)         // Disable create_dir simulation
    )
    .finish();

// Performance-critical: no simulation overhead
let op = Operator::new(S3::default())?
    // Don't add SimulateLayer - use native capabilities only
    .finish();

§For binding users (Python/Java/Ruby)

Binding users will see no breaking changes during the migration. Simulations will continue to work automatically:

# Python - works the same way
operator = opendal.Operator("fs", root="/tmp")
files = list(operator.list("", start_after="file.txt"))  # Just works

§Migration path

§Phase 1: Next version (Soft transition - 0-3 months)

  • SimulateLayer is introduced as a public layer
  • CompleteLayer remains internal, controlled by auto-simulate feature (enabled by default)
  • Documentation encourages explicit use of SimulateLayer
  • No breaking changes
# Still works with auto-simulate enabled by default
[dependencies]
opendal = "0.x"

§Phase 2: Next 2-3 versions (Transition period - 3-6 months)

  • Deprecation warnings added for implicit simulation
  • Migration guide published
  • Examples updated to use explicit SimulateLayer
  • Users can test with default-features = false to disable auto-simulate
# Opt-out of auto-simulate to prepare for migration
[dependencies]
opendal = { version = "0.x", default-features = false }

§Phase 3: Next minor version after transition (Breaking change - 6 months)

  • auto-simulate feature disabled by default
  • Users must explicitly add SimulateLayer or opt-in with features = ["auto-simulate"]
// Required after this version
let op = Operator::new(Fs::default())?
    .layer(SimulateLayer::default())
    .finish();

§Phase 4: Next major version (Clean slate - 12 months)

  • CompleteLayer and auto-simulate feature completely removed
  • Bindings automatically apply SimulateLayer internally
  • Core library requires explicit simulation

§Reference-level explanation

§Architecture changes

§Current architecture

Backend → CompleteLayer (auto-applied) → User
          ↓
          • list recursive simulation
          • stat dir simulation  
          • create_dir simulation
          • reader/writer wrappers
          • (all forced, no user control)

§New architecture

Backend → [SimulateLayer (user-controlled)] → User
          ↓
          • list recursive simulation (configurable)
          • list start_after simulation (configurable)
          • stat dir simulation (configurable)
          • create_dir simulation (configurable)
          • reader/writer wrappers (kept in CompleteLayer or removed)

§SimulateLayer implementation

// core/src/layers/simulate.rs

#[derive(Debug, Clone)]
pub struct SimulateLayer {
    list_recursive: bool,
    list_start_after: bool,
    list_limit: bool,        // Future
    stat_dir: bool,
    create_dir: bool,
}

impl Default for SimulateLayer {
    fn default() -> Self {
        Self {
            list_recursive: true,
            list_start_after: true,
            list_limit: false,
            stat_dir: true,
            create_dir: true,
        }
    }
}

impl SimulateLayer {
    /// Enable/disable recursive listing simulation (default: true)
    pub fn with_list_recursive(mut self, enabled: bool) -> Self {
        self.list_recursive = enabled;
        self
    }
    
    /// Enable/disable start_after simulation (default: true)
    pub fn with_list_start_after(mut self, enabled: bool) -> Self {
        self.list_start_after = enabled;
        self
    }
    
    /// Enable/disable stat directory simulation (default: true)
    pub fn with_stat_dir(mut self, enabled: bool) -> Self {
        self.stat_dir = enabled;
        self
    }
    
    /// Enable/disable create_dir simulation (default: true)
    pub fn with_create_dir(mut self, enabled: bool) -> Self {
        self.create_dir = enabled;
        self
    }
}

§Feature flag for migration

// core/src/types/operator/builder.rs

impl<A: Builder> Operator {
    pub fn new(accessor: A) -> OperatorBuilder<impl Access> {
        let builder = OperatorBuilder { accessor }
            .layer(ErrorContextLayer);
        
        #[cfg(feature = "auto-simulate")]
        let builder = builder.layer(CompleteLayer);
        
        builder.layer(CorrectnessCheckLayer)
    }
}

// Cargo.toml
[features]
default = ["auto-simulate"]  # Next version to Phase 3
auto-simulate = []

§Capability updates

Services that don’t natively support certain features will have their full_capability updated when SimulateLayer is applied:

impl<A: Access> Layer<A> for SimulateLayer {
    fn layer(&self, inner: A) -> Self::LayeredAccess {
        let info = inner.info();
        
        info.update_full_capability(|mut cap| {
            if self.list_start_after && cap.list {
                cap.list_with_start_after = true;
            }
            if self.list_recursive && cap.list {
                cap.list_with_recursive = true;
            }
            if self.create_dir && cap.list && cap.write_can_empty {
                cap.create_dir = true;
            }
            cap
        });
        
        SimulateAccessor { 
            inner: Arc::new(inner), 
            info, 
            config: self.clone() 
        }
    }
}

§Bindings integration

Bindings will automatically apply SimulateLayer to maintain backward compatibility:

// bindings/python/src/operator.rs

impl Operator {
    pub fn new(scheme: &str, options: HashMap<String, String>) -> PyResult<Self> {
        let op = ocore::Operator::new(accessor)?
            .layer(ocore::layers::SimulateLayer::default())  // Auto-apply in bindings
            .finish();
        Ok(Self { core: op })
    }
}

§List start_after simulation

As a concrete example, here’s how start_after will be simulated:

pub struct StartAfterLister<L> {
    inner: L,
    start_after: String,
    skipped: bool,
}

impl<L: oio::List> oio::List for StartAfterLister<L> {
    async fn next(&mut self) -> Result<Option<oio::Entry>> {
        loop {
            let Some(entry) = self.inner.next().await? else {
                return Ok(None);
            };
            
            // Skip entries until we find one > start_after
            if !self.skipped {
                if entry.path() <= self.start_after.as_str() {
                    continue;
                }
                self.skipped = true;
            }
            
            return Ok(Some(entry));
        }
    }
}

§Drawbacks

§Breaking change risk

Even with a phased migration, changing from implicit to explicit simulation is fundamentally a breaking change. Users who don’t follow the migration guide will face compilation errors after Phase 3.

Mitigation:

  • 6-month transition period with clear deprecation warnings
  • Comprehensive migration guide with automated tools
  • auto-simulate feature for temporary backward compatibility

§Documentation burden

This change requires updating extensive documentation, examples, and tutorials across all bindings.

Mitigation:

  • Automated documentation generation
  • Version-specific migration guides
  • Community support during transition

§Potential performance regression for some users

Users who upgrade without understanding the change might add SimulateLayer::default() everywhere, even for backends that don’t need it, adding unnecessary overhead.

Mitigation:

  • Clear documentation about which backends benefit from simulation
  • Performance recommendations in the migration guide
  • Compiler warnings for unnecessary simulations (future enhancement)

§Rationale and alternatives

§Why this design?

  1. Explicit over implicit: Users should consciously decide what simulations to use, leading to better understanding and fewer surprises.

  2. Gradual migration: The phased approach minimizes disruption while giving the community time to adapt.

  3. Binding compatibility: Automatic application in bindings ensures that non-Rust users aren’t affected.

  4. Extensibility: The method chaining pattern makes it easy to add new simulations (like list_with_limit) without breaking changes.

§Alternative 1: Keep CompleteLayer, add disable flags

Instead of a new layer, add flags to disable specific simulations in CompleteLayer.

Rejected because:

  • Doesn’t address the “all-or-nothing” problem fundamentally
  • Still forces all users to pay for simulation machinery even if disabled
  • Doesn’t improve code organization or maintainability

§Alternative 2: Service-specific default simulation

Each service declares which simulations it needs, applied automatically.

Rejected because:

  • Less transparent - users don’t know what’s being simulated
  • Service maintainers must decide defaults, which may not match user needs
  • Doesn’t allow per-application customization

§Alternative 3: Split into multiple layers

Create separate layers: ListSimulateLayer, StatSimulateLayer, CreateDirSimulateLayer.

Rejected because:

  • Too granular for most users
  • Verbose: .layer(ListSimulateLayer).layer(StatSimulateLayer)...
  • Doesn’t align with OpenDAL’s “batteries included” philosophy
  • SimulateLayer with method chaining provides the same granularity when needed

§Alternative 4: Builder pattern

Use a separate builder type like SimulateLayer::builder().with_xxx().build().

Rejected because:

  • More verbose than necessary for simple use cases
  • Method chaining on the layer itself is more ergonomic
  • OpenDAL already uses method chaining in many places (e.g., OpList)

§What if we don’t do this?

  1. CompleteLayer continues to grow in complexity
  2. New simulations (like start_after for fs) remain unimplemented
  3. Performance-sensitive users continue to be frustrated by forced overhead
  4. OpenDAL becomes harder to maintain and understand

§Prior art

§Rust ecosystem

  • Tower (HTTP middleware): Uses explicit middleware with builder pattern
  • Actix-web: Middleware are opt-in with clear APIs
  • Reqwest: Features are controlled via Cargo features, not runtime

§Other storage libraries

  • AWS SDK: Capabilities are documented but not abstracted
  • GCS Client: No simulation layer - users must handle missing features
  • MinIO SDK: Provides compatibility helpers as separate modules

OpenDAL’s approach is unique in providing transparent simulation, and this RFC aims to maintain that advantage while adding user control.

§Unresolved questions

§When to remove reader/writer wrappers?

CompleteLayer currently includes CompleteReader and CompleteWriter that validate read/write sizes. Should these:

  1. Stay in a minimal CompleteLayer (still auto-applied)?
  2. Move to SimulateLayer?
  3. Move to a separate ValidateLayer?

Proposed resolution: Keep in minimal CompleteLayer during transition, revisit in next major version.

§Should we provide migration tooling?

Should we build cargo-opendal migrate or similar tools to automatically update code?

Proposed resolution: Defer to Phase 2, assess based on community feedback.

§Binding-specific configuration?

Should Python users be able to configure SimulateLayer through binding APIs?

# Hypothetical API
operator = opendal.Operator(
    "fs", 
    root="/tmp",
    simulate={"list_start_after": True, "stat_dir": False}
)

Proposed resolution: Not in initial implementation. Can be added in future based on demand.

§Future possibilities

§Automatic simulation detection

Add compile-time or runtime warnings when SimulateLayer is used with backends that don’t need it:

// Future enhancement
let op = Operator::new(S3::default())?
    .layer(SimulateLayer::default())  // Warning: S3 supports all features natively
    .finish();

§Conditional simulation based on operations

Allow enabling simulations only for specific operations:

// Future API
SimulateLayer::default()
    .simulate_only_for(&["list", "stat"])

§Simulation statistics

Expose metrics about which simulations are being used:

// Future API
let stats = op.simulation_stats();
println!("start_after simulated: {} times", stats.start_after_count);

§List limit simulation

Extend simulation to list_with_limit for backends that don’t support it natively (requires client-side buffering).

§Versioning simulation

Extend simulation to list_with_versions for backends that don’t support versioning natively (requires metadata storage).

§Plugin system for custom simulations

Allow users to provide custom simulation logic:

// Far future API
SimulateLayer::default()
    .with_custom_simulator(MyCustomSimulator::new())

§Zero-cost simulation layer

Use const generics or compile-time feature detection to make simulation truly zero-cost when all features are native:

// Speculative future optimization
let op = Operator::new(S3::default())?
    .layer(SimulateLayer::default())  // Optimized away at compile time for S3
    .finish();