Golden Dataset Mcp

io.github.nipunkhanderiaPublished by GitHub account nipunkhanderiav0.1.2Updated Oct 24, 2026

Version-controlled golden datasets and RAG evaluation, no API key needed.

VerifiedSTDIODesktop onlyDeveloper ToolsData & AnalyticsAI & ML

Overview

AI-generated overview

Manages version-controlled golden Q&A datasets and scores RAG/LLM answers with TF-IDF similarity, all from an assistant conversation.

What you can do
  • Create and manage datasets: initialise a dataset at a path, add, edit and delete question/answer entries, and list working-tree or committed entries
  • Version control: snapshot the working tree as an immutable version, diff two versions, and show current version, size and history
  • Evaluate: score actual answers against a committed version using TF-IDF cosine similarity, returning average similarity and per-entry pass/fail
  • Every tool takes an explicit dataset path, so one server instance can handle several datasets without session state
When to use it
Useful when you maintain a golden question/answer set for a RAG or LLM pipeline and want an assistant to add cases, commit versions and compare releases. Also fits automated evaluation runs where answers are scored without calling an LLM.
Requirements
Runs locally over stdio as a Python package installed with pip, which pulls in golden-dataset-studio and scikit-learn. No API key, environment variables or configuration are needed. Datasets are stored as JSON/JSONL files under the given dataset path.
Before you install
Tools write, edit and delete entries and create versions on the local filesystem, so point dataset paths at the intended directory. Evaluation is lexical TF-IDF overlap rather than deep semantic scoring, and very short or stop-word-only answers can make scoring fail. No remote storage or syncing is performed.

Installation

In SourceWeft

  1. Open Golden Dataset Mcp in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

mcp-name: io.github.nipunkhanderia/golden-dataset-mcp

golden-dataset-mcp

An MCP server wrapping golden-dataset-studio — version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines.

This is a thin protocol layer over the existing golden_dataset library (DatasetStore, Evaluator). It does not reimplement any logic — it exposes the library's existing Python API as MCP tools so an agent (Claude Desktop, Claude Code, or any MCP client) can manage golden datasets conversationally.

No LLM API key required. Evaluation uses TF-IDF cosine similarity (scikit-learn), not an LLM call.

Why a separate package from golden-dataset-studio?

golden-dataset-studio is a CLI tool — designed for a human typing golden add, golden commit, etc. in a terminal. golden-dataset-mcp exposes the same underlying operations as MCP tools so an LLM agent can drive them programmatically, e.g. as part of an automated RAG evaluation pipeline. Keeping them as separate PyPI packages means CLI users aren't forced to pull in fastmcp as a dependency, and MCP users get a clean, protocol-focused package.

Tools

ToolWhat it does
init_datasetInitialise a new dataset at a given path
add_entryAdd a question/answer pair to the working tree
update_entryEdit fields of an existing working-tree entry
delete_entryRemove an entry from the working tree
list_entriesList working-tree or committed-version entries
commit_versionSnapshot the working tree as a new immutable version
diff_versionsShow entries added/removed/changed between two versions
evaluate_answersScore actual answers against a version via TF-IDF cosine similarity
dataset_statusShow current version, working tree size, and version history

Design: every tool takes an explicit dataset_path

Unlike the CLI (which operates on the current working directory), every tool here requires an explicit dataset_path parameter. This keeps the server fully stateless between calls — no hidden "current dataset" session state to lose track of, and safe for one server instance to manage multiple datasets or serve multiple concurrent clients.

Installation

bash
pip install golden-dataset-mcp

This pulls in golden-dataset-studio and scikit-learn automatically as dependencies.

Usage with Claude Desktop / Claude Code

json
{  "mcpServers": {    "golden-dataset": {      "command": "golden-dataset-mcp"    }  }}

No environment variables needed — no API key, no config.

Example flow

1. init_dataset(dataset_path="./my-rag-eval", name="support-bot-eval")2. add_entry(dataset_path="./my-rag-eval", question="...", answer="...")   [repeat for each golden Q&A pair]3. commit_version(dataset_path="./my-rag-eval", description="initial 50 questions")4. [run your RAG pipeline, collect actual answers]5. evaluate_answers(dataset_path="./my-rag-eval", actual_answers=[...])   -> avg_semantic_similarity, per-entry scores, pass/fail

As your RAG pipeline changes over time, commit_version again after edits and use diff_versions to see exactly what changed in your golden set between releases.

Relationship to the underlying library

golden-dataset-studiogolden-dataset-mcp
InterfaceCLI (golden ...)MCP tools
Driven byA human typing commandsAn LLM agent / MCP client
Path handlingCurrent working directoryExplicit dataset_path per call
Dependency direction—Depends on golden-dataset-studio

If you want the human-driven CLI, use golden-dataset-studio directly. If you want an agent to drive it, use this package.

Development

bash
git clone https://github.com/nipunkhanderia/golden-dataset-mcpcd golden-dataset-mcppip install -e ".[dev]"pytest -v

Validate the MCP-facing contract:

bash
npx @modelcontextprotocol/inspector golden-dataset-mcp

Limitations

  • evaluate_answers uses TF-IDF cosine similarity, which captures lexical overlap better than deep semantic meaning. For embedding-based or RAGAS-style metrics, call the underlying library's Evaluator.ragas_evaluate() directly (requires pip install "golden-dataset-studio[ragas]" — not exposed as an MCP tool in this version).
  • Very short or stop-word-only answers will raise an error. scikit-learn's TF-IDF vectorizer raises ValueError: empty vocabulary on inputs like a bare "4" or "the a an". Avoid single-token golden answers, or expect evaluate_answers to fail on them.
  • All state is filesystem-backed JSON/JSONL under <dataset_path>/.golden_dataset/; this server does no remote storage or syncing.

License

MIT

Source: README.md at commit 8c58469

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.1.2LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts