Datris

List all registered pipeline configurations. Each pipeline defines a complete data processing flow: source format and schema, AI-powered data quality and transformations, and destination (database, message queue, or vector store). CALL THIS FIRST on any data-related user request — before suggesting external sources, before asking scope questions. The user almost always cares more about what's already in their Datris environment than about a generic options menu.

Takes no parameters.

Get a specific pipeline configuration by name. Returns the full JSON config including source, dataQuality, transformation, preprocessor, and destination sections.

Create a pipeline. FOUR destination categories: STRUCTURED (postgres, mongodb, snowflake, databricks, sqlserver) — send a TINY sample (via content_text) and the schema is auto-detected; snowflake additionally REQUIRES credentialsSecret (a Platform secret with account/user/privateKey or password — discover via list_platform_secrets), warehouse, and database; databricks additionally REQUIRES credentialsSecret (a Platform secret with host plus clientId/clientSecret or token) and database (the Unity Catalog name), plus warehouse (the SQL warehouse ID) unless the Databricks secret has a `warehouse` field — in that case omit warehouse and do NOT ask the human for it (list_platform_secrets / get_platform_secret_fields show the field names); sqlserver additionally REQUIRES credentialsSecret (a Platform secret with host plus the fields its authMethod needs) and database, schema defaults to dbo, and it takes a CSV, JSON or XML sample. OBJECTSTORE (objectstore — writes Parquet files, ORC files, or an Iceberg table to MinIO or AWS S3) — same shape as structured (CSV-only, sample required for schema detection), plus objectStore-specific knobs (bucket, prefix, fileFormat, partitionBy, writeMode; keyFields for iceberg+merge; provider+credentialsSecret for S3). VECTOR (pgvector, qdrant, weaviate, milvus, chroma) — no schema; pass ONLY pipeline + destination (and optionally filename for the file-extension hint), no sample at all. LIVE READ (a Live Read pipeline, destination `scratch`) — same sample-required shape as structured, nothing is landed; the run's rows come back on the status rollup (`resultPreview`) and via get_pipeline_result; results expire — use it when the user wants an answer now, a validation result, or a transformed view with no reason to keep the rows (see the KEEP-OR-READ-LIVE RULE). SAMPLE-SIZE RULE: the sample exists ONLY for schema detection — send the header row plus 3-5 representative rows, NEVER a full dataset. Composing hundreds of rows here wastes minutes

Set or clear the catalog grouping label on an existing pipeline or tap. ONLY call this when the user has explicitly asked to organize work under a named catalog. Do NOT call it proactively — catalogs are a user-chosen organizational convention; assigning one for them puts the pipeline/tap into a taxonomy they didn't ask for. Pass exactly one of `pipeline` or `tap` to identify the target. Pass `catalog` to set the label, or omit it (or pass an empty string) to clear it. Survives subsequent re-ingests — `datris ingest` no longer rewrites an existing pipeline's config. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Rename a catalog: every tap and pipeline labelled with `catalog` is relabelled to `new_name` in one server-side operation, and the empty-catalog placeholder moves with it. Call this ONLY when the user has explicitly asked to rename that catalog — never to emulate a rename by calling set_catalog on each item. `new_name` must not already exist: a rename never merges catalogs and is refused if the target exists. To combine two catalogs, move their items with set_catalog. Uncataloged cannot be renamed, and `new_name` must use letters, digits, '_' and '-' only. Items that could not be moved come back under `failed` (the rest still move); report them to the user. API keys whose capabilities are scoped to the old catalog name are listed under `affectedKeys` — they keep the old scope and no longer match the renamed items, so tell the user an administrator may need to update those keys. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Delete a catalog by moving every tap and pipeline in it to Uncataloged and removing the catalog. No tap, pipeline or data is deleted. Call this ONLY when the user has explicitly asked to delete that catalog. Uncataloged itself cannot be deleted. Items that could not be moved come back under `failed` (the rest still move); report them to the user. Deleting a catalog together with its items and their data is available via the API or UI, not through this tool. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Back up a catalog: returns one zip archive of every tap and pipeline in it (definitions, each tap's script and incremental state, referenced validation schemas). Read-only. Secret values are never included; the manifest lists the secret names the catalog needs. Returns `manifest` (catalog, taps, pipelines, requiredSecrets, warnings) first, then `archiveBase64` and `sizeBytes`. Answer questions about the archive from `manifest`; pass `archiveBase64` unchanged to restore_catalog. Uncataloged cannot be exported.

Read-only. The order to enable a catalog's taps in: taps grouped into levels, level 1 first (taps that read no other tap's tables), then the taps that read tables produced by level 1, and so on. A tap's dependencies are its declared `dependsOn` or, when it declares none, the taps the platform inferred from the tables its script queries. Per tap: `hasPassingTest` (last test run succeeded on the tap's current script; a test of an older script does not count), `lastRealRunLanded` (last real run landed records), `blockedBy` (dependencies that have not landed data yet) and `ready`. A tap whose sources have not landed data fails its test, so run the sources first. Taps on a dependency cycle are listed under `cycles`. Use it to answer "what order do I enable these in", e.g. after restore_catalog, where every tap is disabled with no data landed.

Restore a catalog archive produced by export_catalog as catalog `catalog` (the same or a new name). Call this ONLY when the user has explicitly asked to restore and named the archive to use. A restore never merges: it is refused (with the clashing names) when the catalog already exists or any archived tap or pipeline name exists anywhere on this install. mode=overwrite replaces only items already in `catalog` and needs `confirm` equal to the catalog name. It never writes secrets: missing ones come back under `missingSecrets` for the user to create. Restored taps are left disabled; tell the user to test and enable them. Items that could not be restored come back under `failed`; report them. Archived AI scripts are restored as-is unless `regenerate_scripts` is set; `scriptsImported`, `scriptsRegenerated` and `scriptsKept` (a live script left in place; both with the reason) say which were which. The response carries `sourceEnvironment` (where the archive was exported) and `targetEnvironment` (this install), plus a warning when they differ; tell the user. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Start one server-side job that enables a catalog's taps in their enable order (see catalog_enable_order): level by level, each tap is tested if it has no passing test, enabled, given one real run, and the job waits until that run's data has landed before moving on. A tap whose source did not land data is skipped with the reason; every tap's outcome is reported. Call this ONLY when the user has asked to enable the catalog's taps. It returns at once with the job id and the plan; follow progress with catalog_enable_order (its `job` block), pausing with `wait_seconds` between polls, until `job.active` is false, then report each tap's phase and reason. While the job runs, do not call run_tap, test_tap or update_tap on that catalog's taps: the job is already running them. Refused while another enable job for the catalog is running, or when its taps have a dependency cycle (the `cycles` are returned). Cancelling is done by the user in Datris. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Delete a pipeline. This is DESTRUCTIVE: by default it removes BOTH the pipeline configuration AND all data already written to the destination (MongoDB collection rows, Postgres table rows, vector-store entries). It also wipes document-tap ledgers and staged files for any tap that targets this pipeline, so a recreate gets a clean re-ingest. The platform deliberately does NOT support deleting just the config and orphaning the data — that creates ghost state. If you want to keep the config but wipe the destination data ("reset"), pass keep_config=true; the config survives, the data does not. ALWAYS confirm with the user before calling this tool. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Upload data to a registered pipeline for processing. Send the ENTIRE file content as a single base64-encoded string in ONE call — do not pre-split or chunk the content client-side. Vector destinations (pgvector, qdrant, weaviate, milvus, chroma) apply recursive chunking server-side using the pipeline's configured chunkSize/chunkOverlap; for those, one upload_data call yields many embedded chunks automatically. The pipeline's rules are applied: schema validation, data quality checks, transformations, then routing to the configured destination. A Live Read (`scratch`) pipeline validates or transforms the file and hands the rows back (status rollup `resultPreview`, then get_pipeline_result) instead of landing them. Returns a pipelineToken for tracking job status via get_job_status. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Get job status for an upload_data submission. Pass `pipeline_token` (returned from upload_data) for the recommended path. Pass `pipeline_name` instead for a paginated summary of recent jobs for that pipeline. When queried by `pipeline_token`, the response is `{rollup: {allDone, status, jobs: [...]}, events: [...]}` — poll every few seconds until `rollup.allDone` is true, then read `rollup.status` (`success` | `warning` | `error`) for the outcome. Per-job detail is in `rollup.jobs[]` with `pipelineToken`, `pipeline`, `filename`, `status`, `startedAt`, `lastEventAt`, `elapsed`, and `lastError` (populated on failure with `processName` and `description`). `events[]` is the raw begin/info/end audit trail; the rollup is the source of truth for completion. A failed run's `description` gives the error and its causes in plain words; the full stack trace is not included here (it is in the server log under the run's token and in the Ops activity view). When queried by `pipeline_name`, the response is a paginated array of summary rows; the most recent job is index 0 and its `status` field is `success` | `processing` | `error`. Do NOT proceed to query/search until the job is in a terminal state.

Kill a running pipeline job by its pipeline token. The job thread will be interrupted and the job marked as cancelled. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Send data and use AI to generate a comprehensive data profile: summary statistics per column, data quality issues detected, and suggested validation rules. Use the suggested aiRule when building a pipeline's dataQuality section. The server may withhold row values from the model (DATRIS_AI_SAMPLE_VALUES=false): the profile is then built from column statistics or structure only, sampleValues are empty, and the result carries valuesWithheld: true. If the AI model declines the request (a provider safety decline), the result carries aiDeclined: true and holds the server's column statistics only, with empty qualityIssues and recommendations: read that as no AI review, not as a clean file.

Propose real column types for a pipeline whose destination columns are all stored as text (the default for agent-created pipelines). Stateless: samples up to 1000 rows that already landed in the destination, infers types deterministically, and returns per-column evidence — a few sample values, plus (for columns kept as string by a dirty value) the offending value and the type it blocked. Only for postgres, snowflake, databricks, and sqlserver destinations (on sqlserver, `_json`/`_xml` document columns are never retyped); `eligible: false` with a `reason` of destination-not-supported, already-typed, or no-landed-rows (run the pipeline once first) otherwise. Nothing is stored or changed by this call.

Suggest per-field protection (hmac, mask with an optional preserve, redact, drop, or none, each with a one-line reason) for a pipeline's source fields, computed by the configured CodeGen model from field NAMES and TYPES ONLY: no row value is sent to the model. Stateless: nothing is saved or changed. Pass `pipeline` (an existing pipeline name; its stored source schema is used) or `fields` (an array of {name, type}, e.g. the schema you are about to create). A field that already carries `protect` shows it as `current` next to the suggestion. Show the suggestions to the user and ask which to accept before passing any of them as the `protect` map to `create_pipeline`; never apply them on your own. With `preset` (delimited sources only) the server's rule-based preset classifier is used instead of the model: each field gets an identifier class, a method and a reason, and the result lists the fields it did not recognise and review notes; show all of it, including the review notes, to the user.

Apply destination column types to an all-string pipeline. REQUIRES explicit user approval first. Landed data is migrated before the config changes (postgres retypes in place; sqlserver validates with TRY_CONVERT then retypes in place with ALTER COLUMN in one transaction, leaving `_json`/`_xml` document columns as they are; snowflake/databricks validate then swap the table) — the migration locks or replaces the table, so do NOT apply while a run is in flight. Any landed value that will not cast fails the whole apply with the column named and NOTHING changed — then either re-apply with that field set back to string, or fix the data. `fields` must list EVERY destination column (same names as get_dest_types returned) with its intended type; keep a column as-is by passing type string. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Get the Datris server version and the tap budgets in force on this install: `pipelineMaxPayloadMB` (MB, 0 = unlimited), `tapScriptTimeoutSeconds` and `tapRunTimeoutSeconds`, each with a `...Source` field (`env`, `deprecated-alias`, `derived` or `default`). Cheap to call: call it before sizing a large source and compare the estimates with these values, not with the documented defaults (the budget fields are absent on a server older than this feature; then the documented defaults apply). It also returns `environmentLabel` when the operator set `DATRIS_ENV` on the server (absent otherwise).

Takes no parameters.

Check which backend services are up, down, or not configured. Returns the health status of PostgreSQL, MongoDB, MinIO, ActiveMQ, Kafka, and any configured vector databases (Qdrant, Weaviate, Milvus, Chroma, pgvector). Call this before attempting search or query operations to know which services are available.

Takes no parameters.

Run the platform's operational self-check and return a report: Vault token expiry, AI slot secrets complete, embedding model actually loaded, disk usage, the payload staging area (writable, room for one run, no orphaned run directories), component version skew, and (opt-in) whether each AI model answers. Each non-ok check carries a remediation command for the operator; nothing is changed. Do NOT call run_doctor as part of the normal workflow — it is slow. Only use it for diagnostics when something fails or the user asks about the deployment's health. Host-level checks (Docker volumes, container env drift) need `datris doctor` on the machine running Docker and are not included here.

Semantic search across a Qdrant vector database collection. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Semantic search across a Weaviate vector database class. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Semantic search across a Milvus vector database collection. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Semantic search across a PostgreSQL pgvector table using cosine distance. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. Use list_postgres_tables with vector_only=true to discover available pgvector tables. For RAG: pass the returned text to ai_answer.

Semantic search across a Chroma vector database collection. Takes a natural language query, generates an embedding, and returns the most similar document chunks with similarity scores. For RAG: pass the returned text to ai_answer with the user's question.

Execute a read-only SQL SELECT query against PostgreSQL. Use the metadata discovery tools (list_postgres_databases, list_postgres_schemas, list_postgres_tables, list_postgres_columns) first to explore available data before constructing queries. Only SELECT is allowed; LIMIT is auto-appended if missing. Queries are cancelled if they run too long, so avoid full-table scans. For approximate row counts — "how many rows / records does table X have", overviews, or summarizing several tables — do NOT run exact `SELECT COUNT(*)` (it scans the whole table and can take many seconds on a large one). Instead read the planner estimate, which is effectively instant regardless of table size: `SELECT reltuples::bigint AS estimate FROM pg_class WHERE relname = '<table>'`. Use exact `COUNT(*)` only when the user explicitly needs an exact count of a known-small or filtered result.

Read rows from a pipeline's objectStore destination (Parquet files, ORC files, or an Iceberg table in MinIO or AWS S3). Pass the pipeline name; the server resolves the bucket, prefix, format, and credentials from the pipeline config — same code path the writer uses, so MinIO and S3 destinations both work. Returns up to `limit` rows as JSON objects keyed by column name, plus the resolved s3a:// path and format for transparency. For an Iceberg table the response also carries snapshotId (a decimal string, not a number) and snapshotTimestamp (ISO-8601) of the snapshot read; both are null for other formats. Returns 0 rows (not an error) when the pipeline has no successful runs yet. This is the right tool when list_pipelines shows objectStore as the destination — query_postgres / query_mongodb / search_* will not work against objectStore data.

Run a read-only query against the Snowflake account a pipeline loads into. Pass the pipeline name; the server resolves the account, credentials, warehouse, and role from the pipeline's config — same connection the loader uses, credentials never leave the server. Only works for pipelines whose destination is Snowflake (database with useSnowflake=true). Allowed statements: SELECT (WITH/CTE), SHOW, and DESCRIBE — LIMIT is auto-appended to SELECTs. Omit `sql` to preview the pipeline's destination table (the 'did my load land?' check). Metadata discovery uses the same tool — there are no separate list_snowflake_* tools: `SHOW DATABASES`, `SHOW SCHEMAS IN DATABASE <db>`, `SHOW TABLES IN SCHEMA <db>.<schema>`, `DESCRIBE TABLE <db>.<schema>.<table>`, or query <db>.information_schema.columns. Queries run on the customer's configured warehouse (which costs them compute) — keep them targeted and let the default LIMIT stand unless the user asks for more.

Run a read-only query against the SQL Server or Azure SQL database a pipeline loads into. Pass the pipeline name; the server resolves the host, database, and credentials (any authMethod) from the pipeline's config — same connection the loader uses, credentials never leave the server. Only works for pipelines whose destination is SQL Server (database with useSqlServer=true). Allowed statements: SELECT and WITH (CTE) only, written in T-SQL. Rows are capped automatically: a statement with a top-level ORDER BY gets OFFSET 0 ROWS FETCH NEXT <limit> ROWS ONLY, any other is wrapped in SELECT TOP (<limit>) — do not add LIMIT (not T-SQL). Omit `sql` to preview the pipeline's destination table (the 'did my load land?' check). Metadata discovery uses the same tool through INFORMATION_SCHEMA views — there are no separate list_sqlserver_* tools: `SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES`, `SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'dbo' AND TABLE_NAME = '<table>'`. Queries run on the customer's server — keep them targeted and let the default limit stand unless the user asks for more.

Run a read-only query against the Databricks workspace a pipeline loads into. Pass the pipeline name; the server resolves the workspace, credentials, SQL warehouse, and catalog from the pipeline's config — same connection the loader uses, credentials never leave the server. Only works for pipelines whose destination is Databricks (database with useDatabricks=true). Allowed statements: SELECT (WITH/CTE), SHOW, and DESCRIBE — LIMIT is auto-appended to SELECTs. Omit `sql` to preview the pipeline's destination table (the 'did my load land?' check). Metadata discovery uses the same tool — there are no separate list_databricks_* tools: `SHOW CATALOGS`, `SHOW SCHEMAS IN <catalog>`, `SHOW TABLES IN <catalog>.<schema>`, `DESCRIBE TABLE <catalog>.<schema>.<table>`, or query <catalog>.information_schema.columns. Queries run on the customer's SQL warehouse (which costs them compute) — keep them targeted and let the default LIMIT stand unless the user asks for more. If the warehouse is stopped, the first query auto-starts it and may take longer.

Query a MongoDB collection with optional filter and projection. Use list_mongodb_databases and list_mongodb_collections first to discover available data. Returns matching documents as JSON.

Ask a question in natural language about data in a PostgreSQL table. The AI generates a SQL query from the question and table schema, executes it, and returns the results. Use this instead of writing SQL manually.

List all PostgreSQL databases available in the Datris platform. Use this as the first step when exploring what data has been ingested into PostgreSQL destinations.

Takes no parameters.

List all schemas in a PostgreSQL database. Schemas organize tables within a database (e.g., 'public', 'analytics'). Use after list_postgres_databases to drill into a specific database.

List all tables in a PostgreSQL schema. Set vector_only=true to show only pgvector embedding tables, or false (default) to show regular data tables. Use after list_postgres_schemas.

List all columns and their data types for a specific PostgreSQL table. Use this to understand table structure before writing a query_postgres SQL query.

List all MongoDB databases available in the Datris platform. Use this as the first step when exploring what data has been ingested into MongoDB destinations.

Takes no parameters.

List MongoDB collections. If database is specified, lists collections in that database. If omitted, lists all collections across all databases in 'db.collection' format.

List all collections in the Qdrant vector database. Use this to discover available collections before running search_qdrant.

Takes no parameters.

List all classes in the Weaviate vector database. Use this to discover available classes before running search_weaviate.

Takes no parameters.

List all collections in the Milvus vector database. Use this to discover available collections before running search_milvus.

Takes no parameters.

List all collections in the Chroma vector database. Use this to discover available collections before running search_chroma.

Takes no parameters.

List all pgvector tables (tables with an embedding column) in PostgreSQL. Use this to discover available collections before running search_pgvector.

Takes no parameters.

Ask the Datris AI to answer a question based on provided context. Ideal for RAG workflows: first retrieve relevant chunks using a search tool (search_qdrant, search_pgvector, etc.), then pass the retrieved text as context along with the user's question to get a synthesized answer.

Upload a configuration file to the Datris platform. Supports 'validation-schema' (JSON Schema files used in pipeline dataQuality schema validation). Send the file content as base64. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Update an AI provider secret in the Datris platform. Use this to configure your AI API keys so Datris can use AI features (data profiling, schema generation, AI transformations, RAG). Only AI-related secrets can be updated: anthropic, openai, azure, grok, ollama, embedding. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

List the names of tap secrets that already exist (secrets tagged _type=tap — both agent-created and human-owned ones surfaced via the UI's Tap Secrets section). ALWAYS call this before create_tap_secret: if a suitable secret already exists, prefer reusing it by passing its name as secret_name to create_tap. Only ask the user to provide credentials when no existing secret covers the need.

Takes no parameters.

Return the FIELD NAMES (keys only — never values) of an existing tap secret. Use this after list_tap_secrets to verify a candidate secret has the keys your tap script will need (e.g. API_KEY, USER_AGENT). Secret values are intentionally NOT returned and are never visible to the agent.

List the names of PLATFORM secrets (all secrets NOT tagged _type=tap — the Platform tab in the UI's Secrets section). These are human-owned credentials for destinations and infrastructure (e.g. S3 destination credentials, Postgres connection, MongoDB connection, embedding/vector-store endpoints). The agent can READ these (to look up their names and field shape via get_platform_secret_fields and reference them in pipeline configs) but cannot create, update, or delete them — those operations are the user's responsibility via the Secrets tab. Use this whenever a pipeline's destination needs a credentialsSecret reference (e.g. objectStore with provider=s3) and you need to discover which secrets the user already has.

Takes no parameters.

Return the FIELD NAMES (keys only — never values) of an existing platform secret. Use this after list_platform_secrets to verify a candidate secret has the keys a destination config requires (e.g. an S3 credentialsSecret must contain accessKey, secretKey, region). Secret values are intentionally NOT returned. If the named secret is tap-tagged, this tool refuses — use get_tap_secret_fields for those.

Create or update a secret for a tap to use. The secret's fields are injected as environment variables into the tap's Python script at runtime. Use this before create_tap when the tap needs credentials (API key, DB password, etc.). FIRST call list_tap_secrets to check whether a suitable secret already exists — if so, reuse it instead of creating a duplicate. By default, fails if a secret with this name already exists — pass overwrite=true to replace it (ask the user first). Agents can only overwrite secrets that were also created by an agent (tagged _type=tap); secrets owned by a human user must be updated via the UI. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Delete a tap secret. Only secrets created by an agent (tagged _type=tap) can be deleted via this tool; secrets owned by a human user must be removed from the Secrets tab. Use this to clean up after a tap is no longer needed. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Create a tap — a fetcher that pulls data from an external source and pushes it into a pipeline. Two kinds: a Python script the platform executes (default — provide a plain-English `instruction` to have AI generate the script, or supply your own `script` directly), or `kind: "http"` — a user-hosted HTTP endpoint (any language) that Datris POSTs the run context to on each run and which responds with the tap envelope; pass `endpoint_url` and see the tap-http-contract doc. HTTP taps run no code on the platform; AI script generation and the DATRIS_PLATFORM_* platform-data callback do NOT apply to them, so keep any tap whose fetch logic reads platform data as a Python tap. For a large columnar or tabular source, `yield` each batch (a pyarrow `RecordBatch`, a pandas `DataFrame`, or a list of dicts) instead of each row — the platform serialises a batch natively, several times faster — and read only the columns the pipeline needs (`columns=[...]`). Yielding one row at a time remains correct for small or API-paged sources. If the user wants the tap to feed a pipeline, pass `target_pipeline` now. Without it, `run_tap` will fetch but not persist (response will show `persisted: false, persistedReason: "no_target_pipeline"`) — you'd then need to call update_tap to wire a pipeline. For a fetch-only tap (the user wants the rows back, not landed), target a Live Read (`scratch`) pipeline — see the KEEP-OR-READ-LIVE RULE in the server instructions. If the user mentioned ANY recurrence (nightly, daily, hourly, every morning, market open, etc.), pass `cron_expression` NOW — the platform's scheduler will run the tap on that cadence automatically. This is the canonical way to make a tap recurring; do NOT respond with shell commands or external schedulers for the user to run themselves. See the SCHEDULING RULE in the server instructions. AFTER creating, call `test_tap` to validate the script BEFORE any `run_tap` or before relying on a scheduled cron run — see the VALIDATION RULE. Setting a

List all taps with their status, target pipeline, schedule, and last run info. CALL THIS FIRST (alongside list_pipelines) on any data-related user request — the user's existing taps often reveal what's already being pulled and where, which short-circuits the entire 'should I suggest an external API?' conversation.

Takes no parameters.

Manually trigger a tap. Executes the script; when a target pipeline is configured, hands records to the pipeline async. The response carries `recordCount`, `publisherToken`, `pipelineTokens`, `persisted`, `persistedReason`, and the script's `logs` — but NOT the records themselves. If you need to preview what the script produces, call `test_tap` instead. BEFORE the first run of a newly-created or newly-updated tap, you MUST have called `test_tap` and seen it succeed. See the VALIDATION RULE in the server instructions. Skipping the test on a fresh script pushes potentially-broken data into the destination — and `run_tap` doesn't return records, so you won't see the breakage from the response. PER-RUN PARAMS — pass a `params` object to drive this run with caller-supplied values (date range, id list, page cursor, etc.). Each key/value becomes an env var the script reads via `os.environ.get('DATRIS_TAP_PARAM_<key>')`. Keys must match `[A-Za-z_][A-Za-z0-9_]*` so they map cleanly onto env var names. Values are stringified; nested objects/arrays are JSON-encoded (script can `json.loads()` them back). Use this for anything that varies per-call — date windows, id lists, page cursors, batch sizes. Do NOT rewrite the tap secret to pass per-run params: secrets are for credentials (API keys, DB passwords); rewriting them on every call clobbers concurrent runs, pollutes audit history, and wastes Vault writes. If the tap script doesn't yet read a particular param, update the script by calling `create_tap` again with the same `name` and a revised `script` (create_tap upserts and replaces the existing script). That's the right shape for parameterized runs. REQUIRED next steps based on the response: • `persisted: true` → load is still running. Call `get_pipeline_status(publisher_token=response.publisherToken)` and poll until `rollup.allDone` is true. Then read `rollup.status` (`success`/`warning`/`error`) and `rollup.jobs[].lastError`. Do not report completion or query the destin

Read pipeline ingestion status. Use this after `run_tap` to watch a tap-submitted load progress. Pass `publisher_token` to see every job the tap run submitted (the recommended option — works for both structured and document taps). Pass `pipeline_token` for a single ingestion job. Exactly one of the two must be supplied. Response shape: `{rollup: {allDone, status, jobs: [...]}, events: [...]}`. Poll every few seconds until `rollup.allDone` is true, then read `rollup.status` (`success` | `warning` | `error`) for the outcome. Per-job detail is in `rollup.jobs[]` — each entry has `pipelineToken`, `pipeline`, `filename`, `status`, `startedAt`, `lastEventAt`, `elapsed`, and `lastError` (populated on failure with `processName` and `description`). `events[]` is the raw begin/info/end audit trail if you need it; the rollup is the source of truth for completion. A failed run's `description` gives the error and its causes in plain words; the full stack trace is not included here (it is in the server log under the run's token and in the Ops activity view).

Read the rows a Live Read (`scratch`) pipeline produced. Only Live Read pipelines (destination scratch) have a result — anything else is a 404 (an error body of `Not Found` / `status: 404` — rather than the endpoint's own `Only Live Read (scratch) pipelines have a result` message, or `Only scratch pipelines have a result` on older servers — for a pipeline you know is Live Read means the server predates Live Read results: report the version mismatch and stop, do not retry). The first rows are already on the `get_pipeline_status` rollup as `resultPreview`, so call this ONLY when the rollup's `resultTruncated` is true and you need the rest. Paging with `offset`/`limit` never re-runs the source — it reads the stored result. `limit` is clamped server-side, so the next offset is always `offset + returnedCount`, never `offset + limit`. Results expire after the retention window, so read them promptly; a 410 means the result is gone and the pipeline must be run again. Pass `publisher_token` (from run_tap) or `pipeline_token` (a single ingestion job); exactly one of the two must be supplied. Response shape: `{records: [...], rowCount, returnedCount, offset, truncated, resultUri, resultExpiresAt}` — `rowCount` is the whole result, `returnedCount` this page.

Delete a tap and its stored script. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Get a tap's static definition: configuration, schedule, target pipeline, and the generated Python script content. This is config-only — it returns the SAME data on every call and tells you NOTHING about run state. Do NOT call this to check whether a run is finished or to poll for completion — repeatedly calling get_tap after run_tap is a bug, the response will never change to reflect ingestion progress. For run status: call `get_pipeline_status(publisher_token=...)` (token comes from the run_tap response or from a get_tap_logs entry). For run history: call `get_tap_logs`.

Get the run history for a tap. Returns the last 50 run log entries sorted by most recent first, including status, record count, duration, errors, logs, and `publisherToken` for each run that submitted records to a pipeline. Works for both manual runs (triggered by `run_tap`) and scheduled runs (triggered by the platform's cron scheduler) — they share the same log. Use this to verify whether a scheduled run fired, whether any recent run's script succeeded, and to recover the `publisherToken` for any run if you didn't keep the original `run_tap` response in context. To verify the actual destination ingestion outcome — not just that the script ran — pick the relevant entry and call `get_pipeline_status(publisher_token=entry.publisherToken)`. The tap log only records what the script did; the publisher token is how you trace a run through to whether the data actually landed in the destination.

For a document tap: return the ledger of discovered documents (URI, filename, status, hashes, first/last seen timestamps). The ledger is what tells the platform which documents have already been processed so re-runs skip unchanged files. Pass 'clear_uri' to delete one entry (forces that document to be re-processed on the next run) or 'clear_all=true' to wipe the entire ledger (forces a full re-scan).

Get a tap's incremental-sync state — the bookmark/cursor its script saved after the last successful run (injected into the next run as the DATRIS_TAP_STATE env var). Returns `{tap, state, updatedAt, updatedBy}`; `state` is null when the tap has never committed state (non-incremental tap, or no successful run yet). Use this to see where an incremental tap will resume from, or to debug why a scheduled tap is re-fetching or skipping a window.

Overwrite or reset a tap's incremental-sync state. Pass `state` (a JSON object) to set the bookmark the next run receives via DATRIS_TAP_STATE — e.g. rewind a cursor so a window is re-fetched. Pass `reset: true` to delete the state entirely: the next run sees no DATRIS_TAP_STATE and does a full first-run fetch. The state shape is defined by the tap's own script (read its code via get_tap to see what keys it expects). Only use this on explicit request or to recover a broken cursor — normal runs manage state themselves, committing it only after a successful run. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Test-run a tap without pushing data to the pipeline. Executes the tap's script and returns results, record count, and any errors. Use this to validate a script before running it for real. By default a test pulls the first 20 records and returns; `recordCount` from a test is the preview size, not what `run_tap` will produce. If the error reports exit code -9 (or 137, or "killed") with no traceback, the script ran out of memory because fetch() built its whole result in memory: rewrite fetch() to yield records one at a time (read the source in chunks / pages) instead of returning a list, then test again. For a large columnar or tabular source, `yield` each batch (a pyarrow `RecordBatch`, a pandas `DataFrame`, or a list of dicts) instead of each row — the platform serialises a batch natively, several times faster — and read only the columns the pipeline needs (`columns=[...]`). Yielding one row at a time remains correct for small or API-paged sources. A test is bounded by `TAP_SCRIPT_TIMEOUT_SECONDS` (default 300 s), which is deliberately shorter than the ceiling a real run gets — a test that times out on a large source does not mean `run_tap` will. A run that timed out returns the script's logs, its `[wrapper] streamed N records` progress lines and a partial record count. Read them first: a script that reached the source and was still streaming records when the timeout hit is healthy and too long, not wrong — do not rewrite it; tell the user how far it got and offer a smaller window via `params` or the operator's timeout setting. After two consecutive failed tests, stop and report the exact error text to the user instead of iterating further; never probe the runner environment or read the wrapper to diagnose. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pau

Update an existing tap's CONFIG without regenerating the script. Change the enabled state, CRON schedule, target pipeline, or description. USE THIS to set or adjust a tap's schedule (`cron_expression`) whenever the user describes a recurrence (nightly, daily, every morning, market open, etc.). The platform's scheduler runs the tap on the cadence you set — no external cron, Airflow DAG, or shell loop is needed (or wanted). See the SCHEDULING RULE in the server instructions. VALIDATION RULE: if you're enabling a `cron_expression` on a tap whose script has NEVER been validated, call `test_tap` FIRST and confirm it succeeds. The cadence-change path is safe for taps that have already run successfully; it is not safe to set a cron on a never-tested script. The platform enforces this: setting `cron_expression` on an untested script returns HTTP 409 with the remedy — save the tap without the cron, call `test_tap` until it succeeds, then call `update_tap` again with the cron. Cadence-only changes on a tap whose current script has passed `test_tap` are accepted. To change the SCRIPT itself, call `create_tap` again with the same `name` and the new `script` or `instruction` — create_tap upserts by name and replaces the existing script. There is no separate script-only update tool. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

List the change HISTORY of a tap's definition, newest first. Each entry has `version`, `createdAt`, `createdBy`, and `changeNote`. The platform snapshots a tap's config + script every time it is created or updated. Read-only. IMPORTANT: an EMPTY result does NOT mean the tap has no version — it means the tap has not been edited since versioning was enabled, so no change snapshots exist yet. The tap's CURRENT version number is the `version` field on the tap itself (from `list_taps` or `get_tap`), which is at least 1 for every tap. Use this tool for 'what changed and when'; use `list_taps`/`get_tap` for 'what version is it on now'. See also `get_tap_version`, `diff_tap_versions`, `restore_tap_version`.

View one historical snapshot of a tap's definition: the full config and the pinned Python script as they were at that version. Read-only; does not change the live tap. Get version numbers from `list_tap_versions`.

Compare two versions of a tap's definition. Returns a server-computed field-by-field config diff and a line-level diff of the script. Read-only. `version` is the newer/selected snapshot; `against` is the baseline to compare it to.

Roll a tap back (or forward) to a prior definition version. This is APPEND-ONLY and SIDE-EFFECTING: it reads snapshot `version` and writes it as a NEW latest version (config + that version's script), preserving the full history — nothing is overwritten or lost. 'Rolling forward' is the same call with a higher version number. Only call this when the operator has explicitly asked to restore/roll back this specific tap to a specific version. It does NOT run the tap — after restoring, report the new version and stop; do not chain into `run_tap` unless separately and explicitly asked. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

List the change HISTORY of a pipeline's definition, newest first. Each entry has `version`, `createdAt`, `createdBy`, and `changeNote`. The platform snapshots a pipeline's full config every time it is created or updated. Read-only. IMPORTANT: an EMPTY result does NOT mean the pipeline has no version — it means it has not been edited since versioning was enabled. The CURRENT version number is the `version` field on the pipeline itself (from `list_pipelines` or `get_pipeline`), at least 1 for every pipeline. Use this tool for 'what changed and when'; use `list_pipelines`/`get_pipeline` for 'what version is it on now'. See also `get_pipeline_version`, `diff_pipeline_versions`, `restore_pipeline_version`.

View one historical snapshot of a pipeline's definition: the full config as it was at that version. Read-only; does not change the live pipeline. Get version numbers from `list_pipeline_versions`.

Compare two versions of a pipeline's definition. Returns a server-computed field-by-field config diff. Read-only. `version` is the newer/selected snapshot; `against` is the baseline.

Roll a pipeline back (or forward) to a prior definition version. APPEND-ONLY and SIDE-EFFECTING: reads snapshot `version` and writes it as a NEW latest version, preserving full history. 'Rolling forward' is the same call with a higher version number. Only call this when the operator has explicitly asked to restore/roll back this specific pipeline to a specific version. Report the new version and stop. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

List the version HISTORY of the script the pipeline runs for its AI data-quality rule or AI transformation (`kind`), newest first. Each entry has `version`, `createdAt`, `createdBy`, `changeNote`, `origin` (save, regenerate, run, restore-version, restore, repository), `model`, `commitSha` (repository-backed scripts), `fingerprintMatchesCurrent` (written for the pipeline's current instruction and source schema) and `isLive` (the script runs now use). Every change to the script is kept; the pipeline's definition version does not change when its script does. Read-only. See also `get_script_version`, `diff_script_versions`, `restore_script_version`.

View one version of a pipeline's AI data-quality rule or AI transformation script: the list fields plus the full `script` text, `fingerprint`, `generatedAt` and `contractVersion`. Read-only; does not change what the pipeline runs. Get version numbers from `list_script_versions`.

Compare two versions of a pipeline's AI data-quality rule or AI transformation script. Returns a server-computed line diff: `script.lines` entries typed `ctx`, `add` or `del`. Read-only. `version` is the newer/selected version; `against` is the baseline.

Make an earlier version of a pipeline's AI data-quality rule or AI transformation script the one the pipeline runs. APPEND-ONLY and SIDE-EFFECTING: the restore writes the chosen text back as a NEW latest version (no model call); the pipeline's definition version does not change. If that text is already the live script, nothing is written and the response carries `noop: true`. Refused when the pipeline's instruction or source schema no longer matches the version (HTTP 422 with the reason; nothing is changed), and refused with HTTP 409 when a repository-backed script file was edited since its recorded commit. Only call this when the operator has explicitly asked to restore this specific script version. Report the new version and stop. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status `pending_approval` with an `approvalId`. Tell the user it is waiting for approval in Datris, then poll `get_approval` with that id (pause with `wait_seconds` between polls) — do not re-issue the action.

Sleep for a fixed number of seconds, then return. Use this to pace polling against long-running platform work (most often `get_pipeline_status` after `run_tap`, or `get_job_status` after `upload_data`) so you do not burn tool calls hammering an endpoint that is still in progress. ALWAYS poll once BEFORE the first wait — many runs finish in 1–5 seconds. Then use exponential backoff between polls: 5s, 10s, 20s, 30s, 60s, 60s, ... (cap at 60s normally, 120s only if the run is genuinely glacial). Reset to a short wait (5–15s) on the next cycle whenever a poll shows new jobs flipped to a terminal state — that means you're close to done. If 80%+ of jobs are terminal, wait ~15s; if only 1–2 jobs remain, wait ~10s. Hard upper bound is 120 seconds per call — for longer waits, call this tool in a loop interleaved with a status check.

Read this Datris instance's agent policy: for each action (e.g. tap:delete, pipeline:update:dest-types) whether an agent may do it on its own (auto), must wait for a person to approve it (approve), or is refused (deny). Call this before a delete, schema migration, or other consequential change to know whether it will run immediately or queue for approval, and tell the user accordingly. Returns {enabled, policy:{actions, overrides, limits}, actions:[all policy-able action keys], pendingCount}. When enabled is false, every action is auto.

Takes no parameters.

List the actions this agent queued for human approval under the agent policy, newest first. Each entry has id, action, resource, state (pending | approved | rejected | expired | executed | failed), createdAt, expiresAt and — once decided — decidedBy and the result. Use `state` to filter (default: all).

Poll one queued approval by the `approvalId` a mutating tool returned with status pending_approval. Returns its current state: pending (still waiting for a person), executed (approved and performed — `resultStatus` / `resultBody` carry the outcome of the original call), failed (approved but the replay failed), rejected, or expired. Poll with `wait_seconds` between calls, backing off to 30-60s; a decision can take minutes or hours, so after a few polls tell the user you will check back and stop polling unless they ask you to wait.

List the platform's recovery-agent incidents, newest first. An incident is opened by the platform itself when a scheduled tap finally fails, a pipeline job errors, a tap goes stale, a pipeline's volume swings, or a source changed its columns after a successful run (schema drift); the recovery agent diagnoses it and — within the agent policy's recovery mode — repairs and verifies it (for taps, re-running or fixing the tap; for pipelines, regenerating the stored AI script; the evidence names the script versions before and after). Each entry carries kind, resource, state (open | diagnosing | proposed | awaiting_approval | executing | verifying | resolved | failed | abandoned), classification, a step-by-step narrative, and — when the platform computed evidence for it — a verdict (pending | pass | fail | needs-human). Use state=open for what is being worked right now. Returns {enabled:false, incidents:[]} when the recovery agent is off.

Read one recovery-agent incident by id — its trigger, classification, proposal, step-by-step narrative, approvals it is waiting on, outcome, and its evidence (before/after profile, diff, blast radius, thresholds, verdict). A pipeline incident's repair is regenerating the stored AI script; the evidence names the script versions before and after (scriptVersions), and revertToScriptVersions is what a failed verification restores. The one-line evidence summary of this evidence (numbers and column names, never values) is also what the incident webhook sends (evidenceSummary). When the operator asks about a failure the platform is already working, explain THIS record instead of re-diagnosing from scratch.

Read the deterministic evidence the platform computed for one recovery-agent incident: the baseline profile of the last successful run before it broke, the profile of the real run after the fix, the diff between them (columns added or removed, type changes, record-count delta, null-rate deltas, distribution shift), the blast radius (downstream pipelines, datasets and catalog tables, and which agent identities queried them when reads are audited), the thresholds in force, each threshold check, and the verdict (pending | pass | fail | needs-human). Computed by platform code, never by a model; values from protected or sensitive-looking columns are never included. Returns an error when the incident has no evidence.

Find datasets by meaning: ranks the pipelines your key can read against a natural-language query over their names, descriptions, tags, catalogs, destination field names, and source hosts. Each hit returns where the data lives, how fresh it is, provenance handles (latest run id, config version, script sha), lineage, and a `howToQuery` hint naming the existing query/search tool with pre-filled arguments. Discovery only — nothing is executed for you: make the query call yourself, then pass its provenance handles to ai_answer as `sources` so the answer carries them.

Resolve a stamped `_datris_run_id` value back to its origin: the pipeline run, the tap run that fed it, the script commit, the pipeline config version, and the declared source. Use it when rows or search results carry `_datris_*` provenance fields (pipelines with provenance stamping enabled). Returns one document walking the whole chain.

Traverse the lineage graph from one node: everything upstream (what feeds it) and/or downstream (what depends on it) of a source, tap, pipeline, dataset, or catalog, with the edges between them, freshness for the pipeline involved, and optionally the most recent recorded runs (what each run read and wrote, per destination). Datasets a pipeline used to land into under an earlier configuration are marked `historical`. Use it for impact analysis before changing or deleting something — 'what is downstream of this tap?' — or to see which datasets a pipeline actually wrote. Node names: a tap or pipeline name; a dataset's `kind:coordinates` id as returned by find_data or list_pipelines lineage; a source host; a catalog name.

Browse what a Databricks Platform secret can already see in Unity Catalog, one level per call: no `catalog` lists catalogs; `catalog` lists its schemas; `catalog` + `schema` lists tables; adding `table` returns its columns (name, type, comment) and tags. Use it before creating a Databricks pipeline to check whether the data already exists. Discover the secret name with list_platform_secrets. Tables carrying the `datris_pipeline` tag were loaded by that Datris pipeline (query them with query_databricks and that pipeline). If tag lookups are not permitted for the secret, `tagsAvailable` is false and browsing still works. The SQL warehouse comes from the `warehouse` argument, else the secret's optional `warehouse` field, else an existing Databricks pipeline using the secret. A stopped SQL warehouse auto-starts on the first call, which can take 5-20 seconds (longer for classic warehouses). Read-only; catalog and schema listings are cached for 5 minutes.