Ragmetric Mcp

io.github.MukundaKattaPublished by GitHub account MukundaKattav0.1.1Updated Oct 24, 2026

RAG retrieval IR metrics: recall@k, hit@k, MRR, NDCG@k, evaluate_batch.

VerifiedSTDIODesktop onlyData & AnalyticsAI & ML

Overview

AI-generated overview

Local MCP server that lets an assistant compute RAG retrieval metrics such as Recall@k, Hit@k, MRR and NDCG@k, including batch evaluation.

What you can do
  • Compute recall@k and hit@k for a retrieved list against relevant doc IDs
  • Compute MRR and NDCG@k with binary relevance
  • Run all four metrics across many queries at once with evaluate_batch and get mean scores
  • Work with plain arrays of string document IDs supplied by the assistant
When to use it
Useful when iterating on a RAG retriever and you want the assistant to score retrieval quality against ground-truth document IDs without setting up a notebook or a Python ML stack.
Requirements
Runs as a local stdio process, typically launched with npx from the npm package @mukundakatta/ragmetric-mcp, so Node.js and network access for the package download are needed. No accounts, API keys or environment variables are declared.
Before you install
Retrieved and relevant document ID lists are passed to the local server process; nothing else is declared. Review the package before installing, since it is fetched and run automatically with npx.

Installation

In SourceWeft

  1. Open Ragmetric Mcp in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

@mukundakatta/ragmetric-mcp

MCP server for RAG retrieval evaluation. Lets an LLM compute IR metrics on demand: Recall@k, Hit@k, MRR, NDCG@k, plus an evaluate_batch convenience that runs all four across many queries.

Tools

ToolReturns
recall_at_k(retrieved, relevant, k)Fraction of relevant items in top k
hit_at_k(retrieved, relevant, k)1.0 if any relevant in top k else 0
mrr(retrieved, relevant)1 / rank of first relevant, 0 if none
ndcg_at_k(retrieved, relevant, k)NDCG@k with binary relevance (log2 discount)
evaluate_batch(queries, k)Mean of each metric across many queries

retrieved and relevant are arrays of doc IDs (strings).

Install — Claude Desktop

json
{  "mcpServers": {    "ragmetric": {      "command": "npx",      "args": ["-y", "@mukundakatta/ragmetric-mcp"]    }  }}

Install — Cursor / Cline / Windsurf / Zed

Same shape per tool. npx -y @mukundakatta/ragmetric-mcp over stdio.

Use case

You're iterating on a RAG retriever. Hand the LLM a JSONL of queries with ground-truth doc IDs, ask it to call evaluate_batch, and it returns the mean metrics in one call. No notebook, no boto3, no scikit-learn.

License

MIT OR Apache-2.0.

Source: packages/ragmetric-mcp/README.md at commit 6fad5bb

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.1.1LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts