Module mz_storage::source::s3

Expand description

Functionality for creating S3 sources

This source is constructed as a collection of Tokio tasks that communicate over local (worker-pinned) queues to send data into dataflow. We spin up a single “downloader” task which is responsible for performing s3 object downloads and shuffling the data into dataflow. The downloader will currently download the entirety of each file and emit a degenerate ByteStream into the dataflow containing a single chunk. This will have to be improved before we release S3 sources to production. Then, for each object source, we spin up another task which is responsible for collecting object names from an object name source and sending that name to the downloader.

+----------------+
| bucket scanner +-                               -------
+----------------+ \---                         -/       \-
+----------------+     \--   +------------+    /           \
| sqs listener   +--------X->| downloader +--->| dataflow  |
+----------------+     /--   +------------+    \           /
       .  .  .  .   /--                         -\       /-
      etc .  .  . --                              -------
       .  .  .  .