Module oneshot_source

“Oneshot” sources are a one-time ingestion of data from an external system, unlike traditional sources, they do not run continuously. Oneshot sources are generally used for COPY FROM SQL statements.

The implementation of reading and parsing data is behind the OneshotSource and OneshotFormat traits, respectively. Users looking to add new sources or formats, should only need to add new implementations for these traits.

• OneshotSource is an interface for listing and reading from an external system, e.g. an HTTP server.
• OneshotFormat is an interface for how to parallelize and parse data, e.g. CSV.

Given a OneshotSource and a OneshotFormat we build a dataflow structured like the following:

            ┏━━━━━━━━━━━━━━━┓
            ┃    Discover   ┃
            ┃    objects    ┃
            ┗━━━━━━━┯━━━━━━━┛
          ┌───< Distribute >───┐
          │                    │
    ┏━━━━━v━━━━┓         ┏━━━━━v━━━━┓
    ┃  Split   ┃   ...   ┃  Split   ┃
    ┃  Work 1  ┃         ┃  Work n  ┃
    ┗━━━━━┯━━━━┛         ┗━━━━━┯━━━━┛
          │                    │
          ├───< Distribute >───┤
          │                    │
    ┏━━━━━v━━━━┓         ┏━━━━━v━━━━┓
    ┃  Fetch   ┃   ...   ┃  Fetch   ┃
    ┃  Work 1  ┃         ┃  Work n  ┃
    ┗━━━━━┯━━━━┛         ┗━━━━━┯━━━━┛
          │                    │
          ├───< Distribute >───┤
          │                    │
    ┏━━━━━v━━━━┓         ┏━━━━━v━━━━┓
    ┃  Decode  ┃   ...   ┃  Decode  ┃
    ┃  Chunk 1 ┃         ┃  Chunk n ┃
    ┗━━━━━┯━━━━┛         ┗━━━━━┯━━━━┛
          │                    │
          │                    │
    ┏━━━━━v━━━━┓         ┏━━━━━v━━━━┓
    ┃  Stage   ┃   ...   ┃  Stage   ┃
    ┃  Batch 1 ┃         ┃  Batch n ┃
    ┗━━━━━┯━━━━┛         ┗━━━━━┯━━━━┛
          │                    │
          └─────────┬──────────┘
              ┏━━━━━v━━━━┓
              ┃  Result  ┃
              ┃ Callback ┃
              ┗━━━━━━━━━━┛

Modules

aws_source

AWS S3 OneshotSource.

csv

CSV to Row Decoder.

http_source

Generic HTTP oneshot source that will fetch a file from the public internet.

parquet

Parquet OneshotFormat.

util 🔒

Utility functions for Oneshot sources.

Structs

StorageErrorX

Experimental Error Type.

Enums

ChecksumKind 🔒

Enum wrapper for OneshotSource::Checksum, see SourceKind for more details.

Encoding

Encoding of a OneshotObject.

FormatKind 🔒

An enum wrapper around OneshotFormats.

ObjectFilter 🔒

ObjectKind 🔒

Enum wrapper for OneshotSource::Object, see SourceKind for more details.

RecordChunkKind 🔒

RequestKind 🔒

SourceKind 🔒

An enum wrapper around OneshotSources.

StorageErrorXKind

Experimental Error Type, see StorageErrorX.

Traits

OneshotFormat

Defines a format that we fetch for a “one time” ingestion.

OneshotObject

An object that will be fetched from a OneshotSource.

OneshotSource

Defines a remote system that we can fetch data from for a “one time” ingestion.

StorageErrorXContext 🔒

Functions

render

Render a dataflow to do a “oneshot” ingestion.

render_completion_operator

Render an operator that given a stream of ProtoBatches will call our worker_callback to report the results upstream.

render_decode_chunk

Render an operator that given a stream of OneshotFormat::RecordChunks will decode these chunks into a stream of Rows.

render_discover_objects

Render an operator that using a OneshotSource will discover what objects are available for fetching.

render_fetch_work

Render an operator that given a stream OneshotFormat::WorkRequests will fetch chunks of the remote OneshotSource::Object and return a stream of OneshotFormat::RecordChunks that can be decoded into Rows.

render_split_work

Render an operator that given a stream of OneshotSource::Objects will split them into units of work based on the provided OneshotFormat.

render_stage_batches_operator

Render an operator that given a stream of Rows will stage them in Persist and return a stream of ProtoBatches that can later be linked into a shard.