semlayer

dev.semlayer由 semlayer.dev 發布v0.7.0b2.1更新於 Nov 7, 2026

Infers a semantic layer from your data warehouse and serves it to agents over MCP.

概覽

AI 產生的概覽

從資料倉儲推斷語意層並透過 MCP 提供給助理,用於問題路由、SQL 檢查和唯讀查詢。

可以做什麼
  • 查看狀態並探索本機可見的倉儲連線,然後載入已建立的語意層
  • 從倉儲建立或取消推斷語意層,並儲存或匯出(dbt)
  • 瀏覽語意層:語意搜尋、領域、資料表、資料表詳情、指標與意圖路由
  • 取得與問題相關的脈絡,檢視待確認問題並記錄接受或拒絕的判定
  • 對照語意層檢查 SQL,並把指標編譯成 SQL
  • 對工作階段的倉儲執行一筆唯讀 SELECT
適用情境
適用於助理需要在命名晦澀、業務規則未宣告的雜亂倉儲上回答業務問題的情境。它旨在減少把已取消訂單計入彙總、扇出連結等無聲錯誤,並讓回答建立在經過複核、可攜的語意層之上。
執行需求
透過 stdio 在本機執行,使用 uvx 從 PyPI 安裝(Python)。需要一個可連線的倉儲(Snowflake、BigQuery、PostgreSQL、DuckDB),設定於環境變數、.env 檔、dbt profile 或連線檔中。ANTHROPIC_API_KEY 為選用:用於描述和 LLM 驗證的連結,僅統計推斷不需它。
安裝前請注意
推斷會讀取使用者倉儲的中繼資料和樣本值,耗時數分鐘並花費使用者自己的 LLM 額度,應先取得確認。憑證只以名稱傳遞,不應在對話中輸入。判定和儲存語意層會寫入專案檔案。匿名遙測預設開啟,可關閉;本機模型回應快取會保存結構描述和列舉解碼。

安裝

在 SourceWeft 中

  1. 開啟 儀表板中的 semlayer,將其新增到工作區。
  2. 為需要使用其工具的對話啟用該服務。

Desktop only,透過 STDIO。 STDIO 服務會啟動本機處理程序,因此需要 SourceWeft 桌面主機。

其他 MCP 客戶端

參照 儲存庫 中的啟動說明。

README

semlayer

The semantic layer that infers itself — skip the quarter of hand-writing dbt YAML. Fair Source (FSL-1.1-ALv2); the format and its schemas are Apache-2.0.

Point it at your warehouse. It profiles every column, discovers the foreign keys nobody declared, decodes the status columns, finds the business rules hiding in your aggregate tables, and writes the whole thing down as a portable semantic layer in an openly licensed format — with confidence, provenance, and lifecycle on every single claim — ready for any AI agent to consume over MCP.

bash
pip install "semlayer[warehouses]"semlayer init snowflake            # generates the minimal-grant setup script# `snowflake` reads the environment; `snowflake:<connection>` reads# ~/.snowflake/connections.toml; `snowflake://user@account/db?role=..` is a URLsemlayer infer snowflake -o layer.yaml \  --context ./docs/ --context ./etl-repo/CLAUDE.md   # optional: your wikis/dictionaries as priorssemlayer review layer.yaml         # accept/reject what the engine inferredsemlayer mcp layer.yaml            # serve it to Claude, Cursor, or any MCP clientsemlayer lint layer.yaml query.sql  # check any SQL (yours or an agent's) against the layersemlayer drift layer.yaml snowflake  # catch schema changes (cron- and CI-friendly)

Or start from an agent instead of a terminal: add the MCP server to Claude Desktop, Claude Code or Cursor and ask it what it can see — it discovers the warehouses configured on your machine and says what to do next, with no layer built yet. See docs/mcp.md.

From a checkout (to develop on semlayer itself): pip install -e ".[warehouses]".

Why

AI agents fail on real warehouses: frontier models solved just 21.3% of Spider 2.0's enterprise-warehouse tasks at publication (vs ~91% on the earlier academic Spider 1.0) — and even today's best agentic scaffolds only reach ~30%. The fix is a semantic layer — but every existing tool (dbt, LookML, Cube, Snowflake semantic views) makes humans write it by hand, and it goes stale the day someone runs an ALTER TABLE.

On our messy-warehouse benchmark (cryptic names, zero declared constraints, hidden business rules), an agent using the inferred layer answers 82% of business questions correctly vs 42% from the raw schema (+95% relative), with or without the layer's SQL linter — and the errors it fixes are the silent kind: raw-schema "total revenue" happily sums cancelled orders; a fan-out join quietly triples a total. Full benchmark, methodology changes included, and where we DON'T help →

What gets inferred

Examples from the test suite
Semantic types + rolessts_cd → status code; tot_amt → monetary measure with sum/avg aggregations; PII flagged recall-first
Keys & joins104/104 undeclared FKs on TPC-DS-style naming, F1 = 1.0 on our messy fixture — with zero of the seeded false-FK traps accepted (statistics alone never auto-include; naming + LLM must corroborate)
Enum decodesC=Completed, X=Cancelled — joined from the decode dimension the engine itself discovered
Business rules"these aggregate tables reconcile only when sts_cd <> 'X'" — found by hypothesis-testing, scoped to revenue metrics (scope: measures), never to counts; inherited by a child fact only when its line amounts provably sum to the parent's total
Grain, SCD2, ratios"one row per ord_id"; cust_mstr is SCD2 with eff_start_dt/eff_end_dt/is_curr_flg and an as-of join rule; avg_tot_amt_per_order = revenue / orders
SQL lintingcheck_sql (MCP) / semlayer lint (CLI): deterministic checks of any SQL against the layer — missing required filters, fan-out sums, deprecated tables, SCD2 without a validity window, hallucinated columns, vacuous correlated subqueries — each with a fix hint
Metrics, routing, deprecationrevenue/count metrics with contract-legal filters; "use ord_hdr, avoid ord_hdr_legacy (superseded)"
Descriptionsevery table + column, LLM-written from evidence via join-graph context propagation, judged 0.82–0.89 correct+useful by an independent model
Your docs as priors--context ingests data dictionaries, wiki exports, CLAUDE.md files — and tells you where they're wrong: doc-vs-data contradictions go to review, never silent override (guide)

Everything lands with confidence (calibrated, with the misses published), provenance (which signals produced it), and lifecycle (inferred → reviewed → certified, plus deprecated/orphaned), governed by a normative consumer contract that makes silent misuse — summing across a fan-out, joining SCD2 at current-row, filtering on guessed decodes — non-conforming, not merely unwise.

Cost & privacy

  • ~$0.70 per 100 tables end-to-end on the cheap model tier, with your own API key (measured cost model). LLM calls are escalation-only; ~80% of columns resolve from statistics alone.
  • Several teams, one surface: each team infers its own schema into its own file; an analyst's hub lists them as members and holds the cross-document relationships, semlayer stitch infers those through the analyst's own connection, and every command takes the hub (docs/sets.md).
  • Tiers: every table carries a tier (core, staging, raw, scratch) inferred from its name and ETL columns; raw and scratch tables are modeling: inventory_only -- listed with their shape, never joined, described, metricised or queued for review column by column. On a 2,000-table synthetic estate that is 62% of tables set aside, the review queue cut 4.4x and foreign-key precision 0.31 → 0.94 (measured).
  • --no-llm: fully deterministic mode, zero API calls, still useful (0.780 typing / 0.834 role across 7 fixtures, against 0.809 / 0.838 with the model) — for orgs where LLM access needs procurement.
  • --no-sample-egress: no value read from a warehouse cell is sent to the LLM, at no net accuracy cost across our 7 fixtures (0.809 typing / 0.842 role, versus 0.809 / 0.838 with samples). Per warehouse it trades rather than being uniformly free — up to a point better or worse. Re-measured 2026-10-03; see docs/cost-model.md.
  • Two knobs for speed, output identical at any value: --llm-workers N (default 4) runs that many model calls at once (typing escalation, foreign-key validation, both Describe passes; bounded by your API rate limit, measured 7.2x at 8) and --query-workers N runs that many foreign-key queries at once in your warehouse (default 4 on a network warehouse, since it spends your compute: measured 3.3x at 4 and 6.7x at 8 on BigQuery; every core on an in-process DuckDB file). --no-llm builds still use --query-workers. (measured)
  • Read-only, minimal-grant: semlayer init generates the grant script; the reader persona is proved unable to write — on PostgreSQL by a test that runs in ordinary CI against a real server (tests/test_postgres.py, no credentials needed), and on Snowflake by a live suite runnable against your own account (tests/test_snowflake_live.py).
  • Model-response cache, on your disk and nowhere else: each model response is cached in semlayer/cassettes under your platform cache directory ($XDG_CACHE_HOME or ~/.cache, ~/Library/Caches on macOS, %LOCALAPPDATA% on Windows), so an interrupted whole-warehouse run resumes without paying for the same prompts twice. The files hold what the model wrote about your schema — descriptions, inferred types, and the enum decodes it returned, which do quote the coded values it was shown (X=Cancelled). No credentials, no result sets, and nothing is sent to us. Switch it off with --no-llm-cache or SEMLAYER_LLM_CACHE=off, move it with SEMLAYER_LLM_CACHE=<dir>, or keep cell values out of the prompts entirely with --no-sample-egress.
  • Telemetry: anonymous usage statistics — command, duration, outcome, warehouse type, bucketed table/column counts, version, OS, whether an infer was started from the CLI or from an agent, and for semlayer mcp which MCP client connected and which of our tools it called (one of seven fixed words for the client, never its own name; tool names only, never their arguments). Never names, SQL, or data. See exactly what is sent with semlayer telemetry --show; opt out with semlayer telemetry --off, or per-shell with SEMLAYER_TELEMETRY=off. (what and why)

Honest scope (beta)

  • Best on messy warehouses. On clean, well-named schemas our benchmark shows raw DDL is already sufficient — we publish that negative result rather than hide it. If your warehouse is tidy TPC-DS, you may not need us.
  • ETL input: dbt (--manifest target/manifest.json) -- reads the model DAG to settle join targets statistics cannot. Catalog comments and --context docs are read as priors on every run.
  • Verify it yourself, free: ANTHROPIC_API_KEY= pytest -q replays recorded model responses, so every accuracy floor and benchmark number above is checkable with no API key and no spend — including from a fork, where CI has no secrets (what those recordings are, and what is still untidy about them). The warehouses behind the numbers, with sources, pinned versions and licences, are listed in docs/warehouses.md.
  • Scale: measured to 50,000 tables, as one file or as one layer per team stitched into a hub; the largest real warehouse run so far is about 2,000. Data VOLUME is not the limit -- 4TB and 45TB are the same architecture. The ladder, and how to make it fast.
  • Warehouses: Snowflake, BigQuery, PostgreSQL, DuckDB (+ Iceberg on S3 via the DuckDB bridge). PostgreSQL also reaches Redshift and other Postgres-protocol engines, whose catalogs differ in places and are untested. Exporter: dbt (losses reported, never silent). LLM: Anthropic API (Bedrock/Vertex routing is next).
  • Not included: hosted service, ontology enrichment (it failed our own ablation gate — receipts), LookML/RDF exporters.
  • The eval harness ships in this repo — fixtures, gold layers, competency questions, benchmark runner. Run our numbers yourself: python fixtures/build.py && pytest tests/ -q.

Beta feedback

Running the beta? File the Beta feedback form. GitHub issues are public and permanent, so the form asks only for numbers you are happy to say out loud — don't paste schema names, role names or the *.report.json the CLI writes next to your output. That file stays on your machine; it records which tables and schemas you scoped the run to, so it is yours, not ours. If you want us to see it, email [email protected] — same address for anything else sensitive.

Layout

spec/ format schema + consumer contract · src/semlayer/ the engine · fixtures/ 9 eval warehouses + golds + CQ suites · docs/ benchmark, cost model, spike reports · ARCHITECTURE.md how it works

Licence

Fair Source, not open source. The engine and CLI are FSL-1.1-ALv2: free for any internal use, including in production and CI, with no fee or key; the one prohibition is offering semlayer to third parties as a competing product or service. Every release converts to Apache-2.0 two years after it is published.

The format specification, its JSON Schemas and the exporters are Apache-2.0 — a semantic layer nobody else can read is worth nothing.

Releases 0.1.0b1 through 0.4.0b5 were published under Apache-2.0 and remain so permanently. LICENSING.md has the paths table, the conversion dates, and policy text for security teams.

來源:README.md,提交 5caa09f

工具

21

status

What this server has, what it can see, and what to do next. Call this first. It answers with the loaded layer (its size, age, and how many claims nobody has reviewed yet), the warehouse connections visible from this machine, and a `next` field saying what to do -- including when there is no layer at all, which is the state a fresh install is in.

無需輸入參數。

discover_connections

Which warehouses are reachable from this machine, by name. Reads DuckDB files in the working directory, warehouse environment variables, `$SEMLAYER_HOME/.env`, dbt's profiles.yml and Snowflake's connections.toml. Names only: no credential is read, no value that could be one is reported, and no connection is opened. Each entry says whether semlayer can open it (`supported`) and what it would need if not (`note`). requested: the warehouse the user NAMED, in their words, when it is not in the list -- "databricks", "our iceberg tables in s3", "redshift". The answer says whether semlayer opens it after all (Redshift is the postgres connection), and if not, the nearest route and what to tell them. Pass it rather than guessing: a wrong guess here is either a promise we cannot keep or a user sent away who did not need to be.

輸入參數requested

load_layer

Load one of the layers `status` listed under `layers`. ref: the `ref` from that list, the connection it is for, or a path to a layer file. Returns the new status. Use it when several layers are on this machine and the user has said which warehouse they mean -- and after it, tell them which one you loaded.

輸入參數ref必填

infer

Build the semantic layer for a warehouse. Ask the user first. This READS THEIR WAREHOUSE -- metadata and sample values -- under the connection given, and on a real warehouse it takes minutes and uses their own LLM key, which costs them money. Ask before calling it, and say which warehouse you are about to model. connection: a uri from `status.connections`, or one the user gave. Omit it to use the session's. question: what the user actually asked. Keep it here: when the layer is ready, `status` reminds you to answer it, so a run that outlives this turn does not lose the point of itself. scope: schema names, comma-separated, to model only those. The way to make a big warehouse finish quickly. wait_seconds: how long to hold this call open (capped at 60). A DuckDB file finishes inside it and comes back `ready`, so you can answer in the same turn. Anything larger comes back `inferring`. use_llm: false for a statistics-only layer -- no descriptions, no decode validation, no cost, and much faster. While this call is open it streams progress notifications, so a client that renders them shows a bar. Nothing will wake you AFTER it returns. Tell the user that, and call `status` when they next speak.

輸入參數scopeuse_llmquestionconnectionwait_seconds

cancel_infer

Stop the inference that is running. Takes effect at the next stage.

無需輸入參數。

context_for_question

START EVERY QUESTION HERE once a layer is loaded. Returns the part of the layer that bears on this question -- the tables it selected, their columns, keys, required filters and metrics, the routing knowledge for choosing between them -- plus `open_questions`: the claims underneath that nobody has confirmed. Write your SQL from this, `check_sql` it, then `run_sql`. Put the open questions under your answer as assumptions.

輸入參數question必填

run_sql

Run ONE read-only SELECT against the session's warehouse. `check_sql` runs first and an error means the SQL is NOT executed -- a query that sums cancelled orders runs perfectly well, so "it ran" was never the standard. Warnings do not block, and come back with the rows because they are about whether the number means what the user thinks. Rows are capped and there is a ten-minute time limit; the cap, the limit and whether rows were truncated are all stated in the result, so you can tell the user what actually applied. Nothing is written, nothing is persisted, and your SQL is never rewritten: a required filter is reported, never silently applied. Refused while an inference is running -- one connection is not shared between the two. Ask again when `status` says `ready`. The result carries `assumptions` -- the unconfirmed claims the answer rests on. Answer the user in words first, then render those under it.

輸入參數sql必填

review_questions

What is uncertain in this layer, as questions you can answer. Pull this when the user asks what is unreviewed or unreliable. Do NOT walk the user through it after inference: the questions that matter are the ones an answer rested on, and `run_sql` hands you those as `assumptions`. tables: comma-separated, to scope it to what you are looking at.

輸入參數limittables

review_question

One question in full: every object it covers, and the evidence.

輸入參數key必填

review_answer

Record a verdict. Takes effect immediately, and says who decided. verdict: `accept`, `reject`, or `set:<value>` -- `review_question` gives the shape `set` takes. decided_by: `agent` if this is YOUR call, `person` if the user said it. Never guess this. You may accept a claim whose accept changes no behaviour; rejecting, setting, and accepting anything that unlocks behaviour are the person's, and you will be told which this is. by: the name to record. For a person, what they told you -- ask once if you do not know, never invent it, and never use your own client's name. For yourself, something recognisable like `claude-code/opus`. exclude: comma-separated object locations the verdict should skip. note: your reasoning, kept on the question whether or not the verdict was recorded. This is how you PROPOSE one you may not decide. If the layer is a set (a hub with members) this is refused and tells the user which terminal command records it, so do not promise them it is kept until you have the result.

輸入參數bykey必填noteexcludeverdict必填decided_by必填

review_decisions

Every verdict in this layer and who stands behind each one. Read `signal`: `agent` means nobody has confirmed it. Say so when an answer rests on one.

無需輸入參數。

save_layer

Write the layer into the user's project so their team has it. The cache is per-machine; a file in the repository is what a colleague reads. From then on this session's verdicts land there. Ask where they want it rather than choosing for them. Refused on a set: a hub and its members are several files the user already chose the paths for, and one file cannot hold them.

輸入參數path必填

export_layer

Write a snapshot of the layer in another tool's format. target: `dbt` today. On request only, and lossy -- the result says what the format could not carry. Nothing keeps it in step with the layer afterwards and you must keep answering from the layer, never from the export.

輸入參數path必填target必填

get_domains

List warehouse domains (table groupings) for navigation.

無需輸入參數。

get_tables

List table summaries, optionally within a domain.

輸入參數domain

table_detail

Full detail for one table: columns, keys, relationships, REQUIRED FILTERS.

輸入參數name必填

get_metrics

List defined metrics (name, measure, aggregation, filter).

無需輸入參數。

route_intent

Which tables to use (and avoid) for an analytical intent.

輸入參數intent必填

check_sql

Lint SQL you wrote against the semantic layer BEFORE running it. Deterministic checks: unknown tables/columns, correlated references, deprecated tables, missing required filters, fan-out aggregates, SCD2 joins without a validity window. Fix every error and warning (each finding carries a fix hint), then re-check.

輸入參數sql必填dialect

compile_metric

Compile a metric to correct SQL (joins, filters, time bucketing). group_by: comma-separated columns. On refusal the response lists legal alternatives — retry with those or ask the user; do NOT hand-assemble SQL for a refused request. scd: "current" or "asof", required when the join path crosses a history table. These are different questions — a customer's segment now, or their segment when they ordered — so ask the user which they meant rather than guessing.

輸入參數scdname必填filterdialectcalendargroup_bytime_endtime_graintime_start

版本歷史

1
  1. v0.7.0b2.1最新Nov 5, 2026

評分

已滿足適用檢查項的 80%

該評分衡量此收錄條目的完整程度,依據註冊中心條目、端點與公開來源計算得出。它不是對伺服器本身的評價,也不表示伺服器安全或目前可用:可連通性是帶日期的檢查結果,評分也不包含安裝量或使用者評價。

  • 近 30 天內可連通25 分
  • 已宣告認證方式10 分

    請在註冊中心條目中宣告憑證請求標頭,或保持端點能夠回應探測,以便判定其認證方式。

  • 已列出工具必需15 分
  • 描述必需15 分
  • 圖示10 分
  • 文件15 分
  • 授權條款10 分

    請在儲存庫或註冊中心條目中宣告授權條款。