Module rfc_6828_core 

Core

• Proposal Name: core
• Start Date: 2025-11-27
• RFC PR: apache/opendal#6828
• Tracking Issue: apache/opendal#6829

Summary

Introduce a crate topology where opendal-core owns types, raw traits, and a minimal set of critical layers; each storage backend lives in its own opendal-service-{backend} crate; non-critical layers move to opendal-layer-{name} crates; and a facade crate opendal re-exports core, services, and layers while wiring dynamic service registration and feature aliases.

Motivation

• Decouple compilation units so users only build the services and layers they need, reducing compile time and dependency footprint.
• Make service delivery and versioning independent, enabling faster backend evolution without touching core.
• Allow layer evolution at its own pace while keeping correctness-critical layers close to core for safety.
• Prepare for future plugin ecosystems (e.g., third-party services/layers published independently) without breaking existing API paths.

Guide-level explanation

As a user you depend on the facade crate opendal as before. Differences:

• To enable a backend, turn on the corresponding feature, for example opendal = { version = "x.y", features = ["service-s3"] }. The feature pulls in opendal-service-s3.
• Critical layers (error context, simulate, completeness, correctness check, capability check, immutable index, HTTP client) are always available via opendal::layers because they live in opendal-core.
• Non-critical layers are opt-in features, e.g., "layer-logging" pulls opendal-layer-logging and re-exports LoggingLayer under opendal::layers::LoggingLayer.
• API surface stays the same: opendal::Operator, opendal::services::S3, opendal::layers::LoggingLayer still work; only crate boundaries change under the hood.

Migration for existing users:

# Old
opendal = { version = "0.55", features = ["services-s3", "layers-logging"] }

# New (facade keeps aliases for one major version)
opendal = { version = "0.56", features = ["service-s3", "layer-logging"] }

Reference-level explanation

Crate layout

• opendal-core
  • Modules: types, raw, blocking, docs, critical layers (ErrorContext, Simulate, Complete, CorrectnessCheck, CapabilityCheck, ImmutableIndex, HttpClient, optional TailCut).
  • No service implementations, no non-critical layers.
  • Exposes OperatorRegistry and Scheme but performs no builtin registrations.
• opendal-service-{backend}
  • Contains the backend’s Builder, Config, accessor implementation, scheme constant, and a register(&OperatorRegistry) helper.
  • Depends only on opendal-core (plus backend-specific deps).
• opendal-layer-{name}
  • Each non-critical layer in its own crate (or small groups when strongly coupled, e.g., oteltrace/otelmetrics).
  • Depends on opendal-core and layer-specific deps.
• opendal facade
  • Depends on opendal-core (mandatory), opendal-layer-* and opendal-service-* as optional deps behind features.
  • Re-exports core::{Operator, Result, ...}, critical layers, and optional services/layers.
  • On build, the facade crate registers enabled services via a single ctor initializer to populate DEFAULT_OPERATOR_REGISTRY.
  • Maintains feature aliases for one major version (e.g., services-s3 -> service-s3, layers-logging -> layer-logging).

Feature matrix

• Core features: keep executor/runtime toggles (executors-tokio, blocking), internal helpers (internal-path-cache, internal-tokio-rt), plus minimal TLS/http knobs.
• Service features: service-{backend} live in facade, map 1:1 to opendal-service-{backend} optional dependency.
• Layer features: layer-{name} live in facade, map 1:1 to opendal-layer-{name} optional dependency; critical layers require no feature.

Registry and Scheme

• Operator::from_uri remains, but core contains no builtin registrations.
• Facade constructs registry at init by conditionally calling each service crate’s register based on enabled features.
• Scheme::enabled() moves to facade to aggregate enabled services; core provides base enum and helper for custom schemes.

Testing and CI

• Core tests remain for types/raw/critical layers.
• Each service crate carries unit tests plus behavior tests triggered via shared harness; a new opendal-behavior test crate can reuse the existing fixtures.
• Each layer crate keeps its own minimal tests; integration tests that combine layers/services run in facade or dedicated smoke suites.
• Workspace-level tasks (cargo clippy, cargo fmt --check) cover all crates; behavior matrix splits by service features to control runtime cost.

Compatibility

• Public API paths stay stable through facade re-exports.
• Feature aliases maintained for one major release; deprecation warnings in docs and cargo feature descriptions.
• Crate names and versions remain under Apache OpenDAL namespace.

Drawbacks

• More crates increase release coordination overhead and CI matrix size.
• Users pinning to core-only without facade must manually enable services/layers they need.
• Potential longer resolve time in dependency solvers due to many optional dependencies.

Rationale and alternatives

• Keeping critical layers in core avoids breaking invariants and reduces dependency churn on safety-sensitive code.
• Per-layer and per-service crates maximize compile-time selectivity and allow independent evolution.
• Alternative considered: one opendal-layers mega-crate; rejected to keep feature granularity and lighter dependency pull for single-layer use.

Prior art

• Rust ecosystem crates like tokio vs tokio-util, tower vs tower-* split utilities from core runtime.
• Storage SDKs (AWS SDK for Rust) split per-service crates to reduce bloat and compile time.

Unresolved questions

• Exact set of critical layers that must stay in core (TailCut, Retry?) may need tuning after performance and safety review.
• Whether to group strongly coupled layers (otelmetrics/oteltrace) into one crate or keep fully separate.
• Behavior test harness packaging: shared crate vs duplicated fixtures.

Future possibilities

• Allow third-party service or layer crates to self-register via cargo features or a plugin registry.
• Provide a minimal facade profile that ships only core + registry glue for embedders.
• Introduce codegen to auto-wire service registrations to shrink manual glue in facade.