Opengate Mcp
io.github.nickjlambPublished by GitHub account nickjlambv0.1.4Updated Oct 25, 2026
Check whether an AI answer is grounded in its context — deterministic, no LLM judge.
Overview
Local MCP server that lets an assistant run OpenGATE's deterministic, gold-anchored checks on whether an AI answer is actually grounded in its source context.
- What you can do
- Verify grounding: check whether answers are supported by the supplied context, with fabrication and abstention checks against gold cases, without an LLM judge. Score related capabilities: claim extraction, verdict accuracy, redaction, simplification and retrieval fidelity, each measured against hand-labelled gold data. Run evaluations: execute the offline suite with no API key, or online scorers against a live system through an adapter. Gate regressions: compare runs to a per-adapter baseline and fail on any drop, and write timestamped scorecards or a self-contained HTML report.
- When to use it
- Useful when building or maintaining RAG pipelines, document-QA tools, or legal and scientific assistants that must justify answers from source material, and you want reproducible grounding scores in an agent loop or CI gate rather than an LLM-judged score.
- Requirements
- Runs as a local stdio process via the npm package, so Node.js and npx are needed. No API key is required for the offline suite; online scorers need a configured adapter or an HTTP endpoint config, and REST-backed systems can interpolate environment variables into that config.
Installation
In SourceWeft
- Open Opengate Mcp in the dashboard and add it to a workspace.
- Enable the server for the chats that should use its tools.
Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.
Other MCP clients
Follow the launch instructions in the repository.
README
Open Grounded AI Testing & Evaluation
Deterministic, gold-anchored verification for evidence-grounded AI — no LLM judge.
[CI] [npm] [npm downloads] [PyPI] [Docker] [Docker pulls] [Glama MCP server] [Listed on mcpservers.org] [node] [License: MIT] [PRs welcome]
Quick start · Architecture · Surfaces · Examples · Roadmap · Contributing · Changelog
Evidence over plausibility. OpenGATE verifies AI systems that must justify every answer from source material — RAG pipelines, document-QA tools, legal and scientific assistants. It answers one question above all: can the system prove its answer from the evidence it was given?
The check is deterministic — no LLM-as-judge, no grader model, no six-point verdict scale. Required facts must be present, every number must trace back to the source, and when the context can't answer, the system must abstain rather than fabricate. Because it's pure logic, it's reproducible, free, and fast enough to run on every answer or gate on every commit.
As AI moves into high-stakes domains, evaluation is becoming as fundamental as automated testing is in traditional software. OpenGATE turns grounding failures into numbers you can track, and gates every prompt, model, or workflow change against a baseline — so reliability can't quietly regress.
Quick start (60 seconds)
No API key needed — the offline suite runs deterministic scorers against the bundled gold set:
Point opengate.http.json at your endpoint and add --online --ci to gate your own system. Full walkthrough: Getting Started.
One check, many surfaces
The same deterministic grounding logic ships wherever your stack lives:
Architecture
[OpenGATE architecture: systems under test connect through a one-file adapter to the deterministic core — gold datasets and system answers feed pure-logic scorers, which produce versioned scorecards; a regression gate compares each scorecard to the baseline on every commit — improved or held deploys, regressed fails the build.]Scorers never talk to a system directly — they reach it through a small adapter, so the methodology travels and only the gold set changes. In the development loop it sits where CI sits: change a prompt, model, or pipeline; the regression gate diffs the new scorecard against the baseline — improved or held deploys, regressed fails the build.
Why not DeepEval?
Use both — evals measure, OpenGATE verifies. General-purpose frameworks like DeepEval and OpenAI Evals evaluate AI systems broadly, usually with an LLM judging the output. OpenGATE verifies the narrower, harder promise: that every answer is grounded in evidence:
- Provenance is first-class — does the cited passage actually exist, verbatim, in the source?
- No LLM judge — scores are deterministic checks against hand-labelled gold, so they're reproducible and free to run in CI; your judgment lives in the gold set, not a grader model's.
- Regression detection is first-class — every run is diffed against a per-adapter baseline; a drop fails the build.
Pair a general framework for broad quality metrics with OpenGATE to gate the grounding.
Core concepts
Gold cases — hand-labelled benchmark cases (datasets/cases/): source text, the claims that should be extracted, the sentences that should not be, and reference snippets with known-correct verdicts. Copy _template.json to add one; format in datasets/SCHEMA.md, labelling rules in datasets/LABELING-GUIDE.md.
Scorers — one module per metric family (src/scorers/):
Offline scorers run with no API key — fast enough for every commit. Online scorers exercise a live system through an adapter.
Scorecards — every run writes results/<timestamp>.json stamped with the git SHA, so any result is reproducible and auditable. Per-model runs carry a run_model label, turning the results directory into a measured comparison (accuracy × hallucination × latency × cost).
Regression gate — --baseline saves a reference; later runs print per-metric deltas (▲/▼ in percentage points) and --ci fails the build on any drop. Baselines are per-adapter, so one system's scorecard can't clobber another's.
HTML report — add --report (or opengate report) for a self-contained dashboard: pass/fail per scorer, deltas vs baseline, every named failure. One file, no server, no dependencies.
Evaluating your own system
An adapter is one file: two base exports — onlineAvailable(), onlineConfigHint() — plus at least one complete capability (e.g. grounding → answer()). Scorers check adapter.capabilities and skip cleanly across the boundary; adapters are validated at load with messages naming every missing export.
For REST-backed systems there's a no-code path: the bundled generic HTTP adapter reads endpoint paths and headers from opengate.http.json (${ENV} interpolation, built-in latency/token capture). Full contract and a minimal skeleton: ADAPTERS.md.
Examples
- Evaluating a NIM-powered RAG agent — builds a RAG agent on an NVIDIA NIM model and gates its answers' grounding with OpenGATE, deterministically and with no LLM judge. Includes a runnable Python notebook (
opengate-grounding) and a Node adapter for the CI gate.
Proven in production
Four PharmaTools products run on OpenGATE in CI — four different capability shapes, one evaluation standard. Run against RefCheckr's gold set, OpenGATE:
- surfaced a silent parse-failure mode affecting ~50% of multi-claim verdicts, eliminated with enforced structured output (→ 0);
- halved passage hallucination (5.8% → 2.4%) by driving a measured production model change — a decision made on numbers, not reputation;
- holds claim extraction at 0.91 F1 with 0.93 recall on the committed baseline (run-to-run spread 0.86–0.94 — the splitter is an LLM), and is currently failing its own gate: 2 known non-claims leak into extraction on most runs, an open splitter issue the gate reports rather than rounds away.
Redacta — redaction capability (proof the methodology isn't QA-shaped)
Redacta wraps the @pharmatools/redacta engine, scored against synthetic UK clinical notes with gold-labelled identifiers. On its first run the eval found two real engine bugs (relation phrases swallowing nested names; apostrophe surnames dropped) — both fixed and confirmed (knownGap_closed: 2), then promoted to gold. Current scorecard: 100% recall on 25 gold identifiers, 0 leaks, no open gaps.
Patiently AI — simplify capability (faithfulness of paraphrase)
Patiently AI exercises faithfulness scoring for text that is paraphrase by design. The eval caught the simplifier dropping safety-critical specifics — an antibiotic dose vanished from a discharge summary (anchor recall 86%). A preservation rule took the next run to 100% anchor recall, 0 dropped facts, 0 fabricated numbers — a per-run measurement, not a guarantee: one frozen exp-2 capture made six days after the fix still contains a correct-but-unsourced reference range, flagged by design (RESULTS.md §6). That tail is why the eval now gates Patiently's backend in CI and re-scores the live service weekly, with fabrications failing on first occurrence.
PubCrawl — retrieval capability (the layer everything else grounds on)
PubCrawl has no model — it exercises retrieval fidelity against hand-verified anchors and structural invariants, catching parser regressions (collapsed author arrays, [object Object] leakage) that would poison every downstream citation. That OpenGATE scores a non-AI system at all is the point: evidence-grounded AI is only as trustworthy as the retrieval beneath it.
Full methodology and model comparison: how RefCheckr is evaluated.
Project layout
Documentation
Contributing
Contributions are welcome — especially gold cases (new domains, citation styles), adapters (connect your system), and scorers (new metric families). See CONTRIBUTING.md — including its support & governance expectations and AI-assisted development disclosure; open an issue to discuss large changes. Interfaces may still shift pre-1.0, and semver will signal breaking changes.
Citing OpenGATE
If you use OpenGATE in research, please cite it — citation metadata is in CITATION.cff (GitHub's "Cite this repository" button uses it).
License
MIT — because evaluation frameworks shouldn't be black boxes. If an evaluation influences deployment decisions, engineers should be able to inspect every scorer, metric, and benchmark.
Source: README.md at commit 1e38453
Tools
0Version history
1- v0.1.4LatestSep 16, 2026
Score
50% of the applicable checks met
How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.
- Reached in the last 30 days25 pts
Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.
- Auth declared10 pts
Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.
- Tools listed15 pts
Answer tools/list without a credential, or ship a package that starts in a sandbox without one.
- DescriptionRequired15 pts
- Logo10 pts
- Documentation15 pts
- License10 pts