Database Adapters¶

Datus Agent supports connecting to various databases through a plugin-based adapter system. This document explains the available adapters, how to install them, and how to configure your database connections.

Overview¶

Datus uses a modular adapter architecture that allows you to connect to different databases:

Built-in Adapters: SQLite and DuckDB are included with the core package
Plugin Adapters: Additional databases (MySQL, Snowflake, StarRocks, etc.) can be installed as separate packages

This design keeps the core package lightweight while allowing you to add support for specific databases as needed.

Supported Databases¶

Database	Package	Installation	Status
SQLite	Built-in	Included	Ready
DuckDB	Built-in	Included	Ready
MySQL	datus-mysql	`pip install datus-mysql`	Ready
PostgreSQL	datus-postgresql	`pip install datus-postgresql`	Ready
StarRocks	datus-starrocks	`pip install datus-starrocks`	Ready
Snowflake	datus-snowflake	`pip install datus-snowflake`	Ready
ClickZetta	datus-clickzetta	`pip install datus-clickzetta`	Ready
Hive	datus-hive	`pip install datus-hive`	Ready
Spark	datus-spark	`pip install datus-spark`	Ready
ClickHouse	datus-clickhouse	`pip install datus-clickhouse`	Ready
Trino	datus-trino	`pip install datus-trino`	Ready
Apache Doris	datus-doris	`pip install datus-doris`	Ready
TiDB	datus-tidb	`pip install datus-tidb`	Ready
Hologres	datus-hologres	`pip install datus-hologres`	Ready
MaxCompute	datus-maxcompute	`pip install datus-maxcompute`	Ready
Oracle	datus-oracle	`pip install datus-oracle`	Ready
GaussDB / openGauss	datus-gaussdb	`pip install datus-gaussdb`	Ready (Linux and macOS)

Installation¶

Built-in Databases¶

SQLite and DuckDB are included with Datus Agent and require no additional installation.

Plugin Adapters¶

Install the adapter package for your database:

# MySQL
pip install datus-mysql

# PostgreSQL
pip install datus-postgresql

# Snowflake
pip install datus-snowflake

# StarRocks
pip install datus-starrocks

# ClickZetta
pip install datus-clickzetta

# Hive
pip install datus-hive

# Spark
pip install datus-spark

# ClickHouse
pip install datus-clickhouse

# Trino
pip install datus-trino

# Apache Doris
pip install datus-doris

# Hologres
pip install datus-hologres

# MaxCompute
pip install datus-maxcompute

# Oracle
pip install datus-oracle

# GaussDB / openGauss
pip install datus-gaussdb

Once installed, Datus Agent will automatically detect and load the adapter.

Configuration¶

Configure database connections under agent.services.datasources in agent.yml:

agent:
  services:
    datasources:
      mydata:
        type: sqlite
        uri: sqlite:///path/to/database.db

Each entry under services.datasources is one logical database connection.

SQLite¶

mydata:
  type: sqlite
  uri: sqlite:///path/to/database.db

DuckDB¶

analytics:
  type: duckdb
  uri: duckdb:///path/to/database.duckdb

MySQL¶

production:
  type: mysql
  host: localhost
  port: 3306
  username: your_username
  password: your_password
  database: your_database

PostgreSQL¶

production_pg:
  type: postgresql
  host: localhost
  port: 5432
  username: your_username
  password: your_password
  database: your_database
  schema: public  # optional, default is public
  sslmode: prefer  # optional, default is prefer

Snowflake¶

warehouse:
  type: snowflake
  account: your_account
  username: your_username
  password: your_password  # Use private_key, or exactly one of password/private_key_file
  # private_key: |
  #   -----BEGIN PRIVATE KEY-----
  #   ...
  #   -----END PRIVATE KEY-----
  # private_key_file: /path/to/rsa_key.p8
  # private_key_file_pwd: optional_key_passphrase
  warehouse: your_warehouse
  database: your_database
  schema: your_schema
  role: your_role  # optional

Snowflake supports password authentication and key-pair authentication. Configure private_key, or exactly one of password or private_key_file when private_key is absent. private_key takes precedence over private_key_file and password; set private_key_file_pwd only when the private key is encrypted. Snowflake uses database and schema; do not set catalog for Snowflake.

MaxCompute¶

maxcompute_data:
  type: maxcompute
  database: ${MAXCOMPUTE_PROJECT}
  endpoint: ${MAXCOMPUTE_ENDPOINT}
  access_key_id: ${MAXCOMPUTE_ACCESS_KEY_ID}
  access_key_secret: ${MAXCOMPUTE_ACCESS_KEY_SECRET}
  # schema: default               # optional; schema-enabled projects only
  namespace_mode: auto            # auto, two_level, or three_level
  # quota_name: ${MAXCOMPUTE_QUOTA_NAME}
  # tunnel_endpoint: ${MAXCOMPUTE_TUNNEL_ENDPOINT}
  # timeout_seconds: 30
  # query_timeout_seconds: 600

database is the MaxCompute project. Each datasource is bound to one project, has no catalog, and does not generate cross-project SQL. Leave schema unset for a two-level project.table project. Schema-enabled projects use project.schema.table; when schema is omitted, the adapter uses default. Keep namespace_mode: auto unless the configured identity cannot probe schema support. endpoint is the MaxCompute service endpoint; configure tunnel_endpoint separately only when Instance Tunnel uses a different endpoint. Advanced execution settings also include timeout_seconds, query_timeout_seconds, and default_hints.

StarRocks¶

analytics:
  type: starrocks
  host: localhost
  port: 9030
  username: root
  password: your_password
  database: your_database

ClickZetta¶

lakehouse:
  type: clickzetta
  service: CLICKZETTA_SERVICE
  username: CLICKZETTA_USERNAME
  password: CLICKZETTA_PASSWORD
  instance: CLICKZETTA_INSTANCE
  workspace: CLICKZETTA_WORKSPACE
  schema: CLICKZETTA_SCHEMA
  vcluster: CLICKZETTA_VCLUSTER

Hive¶

hive_data:
  type: hive
  host: 127.0.0.1
  port: 10000
  username: hive
  database: default
  auth: NONE  # optional: NONE, LDAP, CUSTOM, KERBEROS
  configuration:  # optional Hive session config
    hive.execution.engine: spark

Spark¶

spark_data:
  type: spark
  host: localhost
  port: 10000
  username: spark
  database: default
  auth_mechanism: NONE  # optional: NONE, PLAIN, KERBEROS

ClickHouse¶

analytics:
  type: clickhouse
  host: localhost
  port: 8123
  username: default
  password: your_password
  database: your_database

Trino¶

trino_data:
  type: trino
  host: localhost
  port: 8080
  username: trino
  catalog: hive
  schema: default
  http_scheme: http  # optional: http or https

Apache Doris¶

doris_data:
  type: doris
  host: localhost
  port: 9030
  username: root
  password: your_password
  catalog: internal      # optional, default is internal
  database: your_database
  charset: utf8mb4       # optional, default is utf8mb4
  autocommit: true       # optional, default is true
  timeout_seconds: 30    # optional, default is 30

Option	Default	Description
`host`	`127.0.0.1`	Frontend (FE) host
`port`	`9030`	FE MySQL-protocol query port, not the FE HTTP port (8030)
`username`	required	Doris user
`password`	empty	Doris password
`catalog`	`internal`	Catalog the connection starts in
`database`	none	Database the connection starts in
`charset`	`utf8mb4`	Connection character set
`autocommit`	`true`	Autocommit mode
`timeout_seconds`	`30`	Connection timeout

Doris speaks the MySQL protocol, so port is the FE query port (default 9030). Objects are addressed as catalog.database.table: Doris has no schema level between database and table, so leave schema unset and let catalog carry the extra level.

internal is the built-in catalog holding Doris-managed tables. To start a connection on an external catalog — a Hive Metastore catalog, for example — name it in catalog:

doris_hive:
  type: doris
  host: localhost
  port: 9030
  username: root
  password: your_password
  catalog: hive_catalog
  database: warehouse

The catalog has to exist in Doris already — Datus selects a catalog, it never creates one. Create it on the Doris side first (CREATE CATALOG hive_catalog PROPERTIES (...), with the metastore URI and storage credentials the catalog type needs) and confirm it is listed by SHOW CATALOGS.

Within a session, SWITCH <catalog> changes the catalog and USE [<catalog>.]<database> changes the database. Switching catalogs clears the current database, because a database of that name usually does not exist under the new catalog; issue a USE afterwards to select one.

TiDB¶

tidb_data:
  type: tidb
  host: 127.0.0.1
  port: 4000
  username: root
  password: your_password
  database: your_database
  charset: utf8mb4       # optional, default is utf8mb4
  autocommit: true       # optional, default is true
  timeout_seconds: 30    # optional, default is 30

Option	Default	Description
`host`	`127.0.0.1`	TiDB server host
`port`	`4000`	TiDB's own default SQL port — not MySQL's 3306
`username`	required	TiDB user
`password`	empty	TiDB password
`database`	none	Database the connection starts in
`charset`	`utf8mb4`	Connection character set
`autocommit`	`true`	Autocommit mode
`timeout_seconds`	`30`	Connection timeout

TiDB speaks the MySQL wire protocol and is addressed as database.table: like MySQL it has no schema level, so leave both catalog and schema unset. Set port explicitly — TiDB listens on 4000, and 3306 on the same host is usually a different server entirely.

Being MySQL-compatible does not make TiDB a MySQL superset. It has no FULL OUTER JOIN, JSON_TABLE, LATERAL, CREATE TABLE ... AS SELECT, CORR/COVAR_*, materialized views, or updatable views, and its default collation is the case-sensitive utf8mb4_bin. Two clauses are accepted and then ignored: CHECK constraints are not enforced unless tidb_enable_check_constraint is ON, and FULLTEXT indexes are silently dropped. The adapter ships a SQL skill that keeps generated SQL clear of all of these.

TiFlash. TiDB's columnar replica engine is per table: ALTER TABLE t SET TIFLASH REPLICA 1 and the optimizer starts choosing between row store and columnar on its own — no query change, and nothing to configure in Datus. information_schema.TIFLASH_REPLICA reports which tables have a synced replica. Aggregate window functions are the one thing that tends not to gain from a replica: on TiDB v8.5 they fall back to single-node computation on the TiDB layer — correct, but not parallel — so prefer GROUP BY aggregation where a query can be written either way. Push-down coverage varies by version and by the operators around the call, so confirm with EXPLAIN and look for mpp[tiflash] on the window operator before assuming either behaviour.

The datus-tidb adapter does not support TLS configuration yet, so it cannot reach TiDB Cloud endpoints that require TLS. This is an adapter limitation, not a TiDB one.

Hologres¶

hologres_data:
  type: hologres
  host: your-instance-cn-hangzhou.hologres.aliyuncs.com  # console endpoint, may embed ":80"
  port: 80
  username: ${HOLOGRES_ACCESS_KEY_ID}
  password: ${HOLOGRES_ACCESS_KEY_SECRET}
  database: your_database
  schema: public   # optional, default is public
  sslmode: prefer  # optional, default is prefer

Hologres (Alibaba Cloud) uses the PostgreSQL wire protocol with AccessKey credentials. access_key_id/access_key_secret are also accepted as aliases for username/password. The host accepts either a plain hostname or a hostname:port console endpoint; an explicit port must match an embedded one.

Oracle¶

oracle_data:
  type: oracle
  host: localhost
  port: 1521
  username: datus_user
  password: your_password
  service_name: FREEPDB1
  schema: DATUS_USER

Configure exactly one connection target: service_name (recommended), sid, or dsn. The service/PDB selects the connection target but is not part of an SQL object name; qualify objects as SCHEMA.TABLE.

GaussDB¶

gaussdb_data:
  type: gaussdb
  host: localhost
  port: 5432
  username: datus
  password: ${GAUSSDB_PASSWORD}
  database: postgres
  schema: public   # optional, default is public
  # driver: pg8000  # optional; omit to use gaussdb on Linux or pg8000 on macOS
  sslmode: verify-ca
  sslrootcert: /etc/datus/certs/gaussdb-ca.pem

GaussDB and openGauss speak the PostgreSQL wire protocol. Choose the driver to match the platform and the server account's password authentication:

Driver	Platform	Authentication
`gaussdb` (Linux default)	Linux	sha256, md5, sm3
`pg8000` (macOS default)	Linux and macOS	sha256, md5
`psycopg2`	Linux and macOS	md5 only; escape hatch

All drivers accept disable, allow, prefer (default), require, verify-ca, and verify-full, but their retry behavior is not identical. For production, use verify-ca with sslrootcert as the baseline: the connection is encrypted and the server certificate must chain to that CA. When the configured host is guaranteed to match the server certificate, use verify-full for stricter hostname validation; otherwise keep verify-ca. require encrypts but does not authenticate the server; prefer permits an unencrypted fallback. If the server merely enables TLS, the client does not need a certificate. If it requires TLS, prefer normally negotiates TLS, but explicitly use require or either verify-* mode to prevent a plaintext fallback against a differently configured endpoint. Only verify-ca and verify-full need the server CA configured on the client. The pg8000 path treats allow like prefer (TLS first), because its API cannot express libpq's plaintext-first retry order. The adapter currently supports one-way TLS, not mutual TLS (sslcert/sslkey).

Both centralized and distributed deployments are supported. The connector auto-detects the database's A (Oracle), B (MySQL), or PG compatibility mode so generated SQL follows the server semantics.

Multiple Database Entries¶

agent:
  services:
    datasources:
      source_db:
        type: mysql
        host: source-server
        username: reader
        password: password
        database: source

      target_db:
        type: snowflake
        account: your_account
        username: writer
        password: password
        warehouse: compute_wh
        database: target

Features by Adapter¶

Common Features¶

All adapters support:

SQL query execution (SELECT, INSERT, UPDATE, DELETE)
DDL operations (CREATE, ALTER, DROP)
Metadata retrieval (tables, views, schemas)
Sample data retrieval
Connection pooling and timeout management

Adapter-Specific Features¶

MySQL¶

INFORMATION_SCHEMA queries
SHOW CREATE TABLE/VIEW support
Full CRUD operations

PostgreSQL¶

INFORMATION_SCHEMA queries
Tables, views, and materialized views support
Multi-schema datasource support
SSL connection modes (disable, allow, prefer, require, verify-ca, verify-full)
SQLAlchemy-based (psycopg2 driver)

Snowflake¶

Multi-database and schema support
Tables, views, and materialized views
Arrow format for efficient data transfer
Native SDK integration

StarRocks¶

Multi-Catalog support
Materialized view support
MySQL protocol compatibility

ClickZetta¶

Workspace and schema management
Volume/Stage file operations
Native SDK integration

Hive¶

HiveServer2/Thrift protocol connection
Hive session configuration support
Multiple auth mechanisms (NONE, LDAP, CUSTOM, KERBEROS)
Database context switching (USE statement)

Spark¶

Spark Thrift Server connection via HiveServer2 protocol
Multiple auth mechanisms (NONE, PLAIN, KERBEROS)
Spark SQL dialect support

ClickHouse¶

HTTP protocol connection
No schema layer (databases serve as schemas)
ClickHouse-specific DML syntax (ALTER TABLE UPDATE)
Lightweight deletes support

Trino¶

Three-level hierarchy: catalog → schema → table
Cross-catalog query support
Built-in TPC-H connector for benchmarking
HTTP/HTTPS connection with SSL support

Apache Doris¶

MySQL protocol compatibility on the FE query port
Multi-catalog support: SHOW CATALOGS discovery, plus SWITCH <catalog> and USE [catalog.]database context switching
Catalog-qualified information_schema reads, so metadata queries need no session-level catalog switch and stay thread-safe
Three-part identifiers (catalog.database.table) with backtick quoting; no schema level
Asynchronous materialized views discovered through mv_infos(), with DDL retrieval
Key-model-aware column metadata (Duplicate, Unique, and Aggregate key columns)
Catalog-aware sample-row retrieval, and list, CSV, Pandas, and Arrow result formats
A packaged db-doris-sql skill covering table models, distribution, materialized views, and loading through Stream Load, Routine Load, or INSERT INTO SELECT over a TVF or catalog
Migration target support: table-layout suggestions, DDL validation, source-type mapping, and a dry-run CREATE TABLE against the cluster

Hologres¶

PostgreSQL wire protocol (PostgreSQL-compatible SQL dialect)
Alibaba Cloud AccessKey authentication
Multi-schema datasource support
Console endpoint normalization (hostname or hostname:port)
SSL connection modes (disable, allow, prefer, require, verify-ca, verify-full)

Oracle¶

Oracle Database 19c SQL and PL/SQL syntax guidance through the adapter-provided skill
Service name, SID, or DSN connection targets
Schema-scoped metadata discovery through ALL_* dictionary views
Oracle-compatible profiling and bound-parameter data transfers

GaussDB¶

PostgreSQL wire protocol (PostgreSQL-compatible SQL dialect)
sha256, md5, and sm3 authentication through the official gaussdb driver
Pure-Python pg8000 path with sha256/md5 authentication on Linux and macOS
TLS modes through verify-full; verify-ca is the production baseline, and verify-full adds hostname validation
A / B / PG compatibility-mode auto-detection, so SQL generation follows the server's semantics
Centralized and distributed deployments, with distribution-aware table DDL
Multi-schema datasource support

Troubleshooting¶

Adapter Not Found¶

If you see an error like Connector 'mysql' not found, make sure you have installed the corresponding adapter package:

pip install datus-mysql

Connection Issues¶

Check the following:

Network connectivity: Ensure you can reach the database server
Credentials: Verify username and password are correct
Port: Confirm the correct port is specified
Database name: Ensure the database exists

Driver Dependencies¶

Some adapters require additional system dependencies:

MySQL: Requires pymysql (installed automatically)
PostgreSQL: Requires psycopg2-binary (installed automatically)
Snowflake: Requires snowflake-connector-python (installed automatically)
Hive: Requires pyhive, thrift, thrift-sasl, pure-sasl (installed automatically)
Spark: Requires pyhive, thrift, thrift-sasl, pure-sasl (installed automatically)
ClickHouse: Requires clickhouse-sqlalchemy (installed automatically)
Trino: Requires trino (installed automatically)
Apache Doris: Requires datus-mysql and pymysql (installed automatically)
Hologres: Requires datus-postgresql and psycopg2-binary (installed automatically)
Oracle: Requires oracledb (installed automatically; Thin mode needs no Oracle Client)
GaussDB: Requires datus-postgresql; dependencies are installed automatically. Linux defaults to the official gaussdb driver and the package wheel bundles its GaussDB-family libpq. macOS defaults to the pure-Python pg8000 driver because no compatible native libpq is published for Darwin.

Architecture¶

datus-agent (Core)
├── Built-in Adapters
│   ├── SQLite Connector
│   └── DuckDB Connector
│
└── Plugin System (Entry Points)
    ├── datus-sqlalchemy (Base layer)
    │   ├── datus-mysql
    │   │   ├── datus-starrocks
    │   │   └── datus-doris
    │   ├── datus-postgresql
    │   │   ├── datus-hologres
    │   │   └── datus-gaussdb
    │   ├── datus-hive
    │   ├── datus-spark
    │   ├── datus-clickhouse
    │   ├── datus-trino
    │   └── datus-oracle
    │
    └── Native SDK Adapters
        ├── datus-snowflake
        └── datus-clickzetta

The adapter system uses Python's entry points mechanism for automatic discovery. When you install an adapter package, it registers itself with Datus Agent and becomes available for use.

Next Steps¶

Quick Start Guide - Get started with Datus Agent
Configuration Reference - Detailed configuration options