S3 Plugin

The S3 plugin (datus-s3-plugin) lets the Datus agent browse, query, and move object-storage data. It speaks to Amazon S3 and to S3-compatible stores such as MinIO and Alibaba Cloud OSS, and it is the transfer layer other plugins rely on when a workflow reads from or writes to an s3:// URI — deploying Airflow DAGs, for example.

Installation

datus plugin install datus-s3-plugin

Requires datus-agent >= 0.3.8. See Plugins for other install sources and profile management.

Skills

Skill	Purpose
s3	Browse, read, query, and move objects
s3-setup	Create a profile (region, credentials, default bucket, compatibility mode)

s3

With this skill the agent can:

• browse and read — list buckets and prefixes, show object metadata, print an object's contents or first lines, and mint presigned URLs (a presigned PUT URL grants write access and is treated like a credential);
• query in place — run S3 Select SQL against a single CSV, JSON, or Parquet object without downloading it, e.g. counting rows per group straight off the bucket;
• move data — upload and download files, sync a directory (only new and changed files are transferred), move objects, and delete them;
• publish artifacts safely — the skill encodes upload hygiene: verify an upload by reading its metadata back, publish each build under a versioned key instead of overwriting a mutable one, and never let credentials end up inside uploaded files.

Writes use SSE-KMS when the profile sets kms_key_id. In aliyun-oss compatibility mode, S3 Select and SSE-KMS are rejected explicitly.

Note that AWS no longer offers S3 Select to new customers: only accounts that were using it before July 25, 2024 can still call it. Newer accounts should query through Amazon Athena or download the object and filter locally instead. MinIO continues to support the S3 Select API.

s3-setup

Ask the agent to set up the plugin and the skill collects the region, the credential source (the default AWS chain, a named profile, keys as ${ENV_VAR} references, or a role to assume), an optional default bucket for bare keys, and an optional SSE-KMS key, then verifies the profile with a read-only listing. MinIO needs only an endpoint_url; Alibaba OSS uses the dedicated compatibility fields. A resulting profile looks like this:

agent:
  plugins:
    s3:
      prod:
        default: true
        region: us-east-1
        bucket: my-data-lake  # optional default bucket
        # credentials: standard AWS chain, or profile / keys / role_arn
      minio-local:
        endpoint_url: http://minio:9000
        access_key_id: ${MINIO_ACCESS_KEY}
        secret_access_key: ${MINIO_SECRET_KEY}

The IAM principal needs at least s3:ListBucket and s3:GetObject; listing the account's buckets additionally needs s3:ListAllMyBuckets (its Resource must be "*"). Add s3:PutObject for uploads and s3:DeleteObject for deletes.

Using with the Agent

Some example requests, once a profile exists:

• "What's in s3://my-lake/events/?" — lists the prefix.
• "Show me the first ten rows of s3://my-lake/events/day.csv."
• "How many events per region are in that file?" — an S3 Select query; nothing is downloaded.
• "Sync ./artifacts to s3://my-lake/artifacts/v1/ and verify the upload."
• "Set up the s3 plugin against our MinIO." — runs s3-setup.

Under the permission system, reads, queries, and presigning run without confirmation; uploads, syncs, and moves ask once under the normal mode; deletes always ask.

Orchestrating Workflows

The transfer layer for DAG deployment

The Airflow and MWAA plugins contain no object-storage client on purpose. Whenever a DAG deployment, export, backup, or migration targets an s3:// URI, the agent routes the transfer through this plugin:

• Deploy — "deploy dags/sales_daily.py to prod Airflow": the agent uploads through this plugin into the scheduler's dags_folder (or an MWAA environment's DAG prefix), then verifies through the scheduler-side plugin that the DAG parses.
• Back up — "export all active prod DAGs to s3://backup/airflow/": the Airflow or MWAA export skill collects the sources through the scheduler's API and hands this plugin the upload.

Credentials stay separated: this plugin owns the storage credentials, the scheduler plugins own the Airflow credentials, and neither sees the other's.

Publish pipeline artifacts

The same pattern serves anything a pipeline consumes from object storage — job JARs, rendered configs, exported query results. Ask the agent to publish a directory under a versioned prefix: it syncs only what changed, then reads the uploads back to verify them.

Related Docs

• Plugins — install sources, profiles, activation, and permissions
• Airflow plugin — DAG deployment on self-managed Airflow
• MWAA plugin — DAG deployment on Amazon MWAA
• Skills — how skills are discovered and loaded