Calibration Explorer

io.github.m-sanchezv0.2.0Updated Oct 5, 2026

Check classifier confidence against accuracy, fit temperature scaling and save local reports.

VerifiedSTDIODesktop onlyAI & MLData & Analytics

Overview

AI-generated overview

Lets an assistant check whether a classifier's saved confidence matches its actual accuracy, fit temperature scaling, and write local HTML and JSON reports.

What it does
A local stdio MCP server for Calibration Explorer. Its tools load prediction files, assess whether saved confidence matches observed correctness, fit a single temperature on calibration rows, set and lock an acceptance threshold on policy-validation rows, review test outcomes, and generate HTML and JSON assessment reports. It reuses the same parser, numerical library, workflow guards and report generator as the browser app, and does not train or run a classifier.
When to use it
Use it when you have saved classifier predictions and want to know whether the model's confidence is trustworthy, calibrate it with temperature scaling, or fix an acceptance threshold before looking at test data. It suits local, file-based calibration work inside a project directory.
Requirements
Runs locally over stdio via npx from the npm package @m-sanchez/calibration-explorer-mcp; Node.js 20 or newer is required. The --input-root and --output-root arguments are required and repeatable, and must be absolute existing directories. No accounts, API keys, environment variables or headers are declared. The verified client is the Claude Code interactive terminal; other MCP hosts are not claimed as verified.
Before you install
It reads prediction files you select and writes reports into new folders only; paths outside the configured roots are refused and existing files are never overwritten. Test outcomes stay unopened until review_test receives an explicit decision through the client's elicitation form. There is no telemetry or remote service, but returned summaries can still reach your MCP client's model provider.

Installation

In SourceWeft

  1. Open Calibration Explorer in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

Calibration Explorer MCP

Local stdio MCP server for Calibration Explorer. It checks whether a classifier's saved confidence matches how often it was correct, fits one temperature on calibration rows, evaluates and locks an acceptance threshold on policy-validation rows, and writes HTML and JSON assessment reports into your project.

It reads prediction files you select, uses the same parser, numerical library, workflow guards and report generator as the browser app, and does not train or run a classifier. There is no telemetry or remote service. Returned summaries can still reach your MCP client's model provider.

Use with Claude Code

Save this configuration, replacing both directories with absolute paths that already exist:

json
{  "mcpServers": {    "calibration": {      "type": "stdio",      "command": "npx",      "args": [        "-y", "@m-sanchez/[email protected]",        "--input-root", "ABSOLUTE_PREDICTION_DIRECTORY",        "--output-root", "ABSOLUTE_REPORT_DIRECTORY"      ]    }  }}

Then launch claude --mcp-config calibration.mcp.json --strict-mcp-config and check /mcp. The verified client is the Claude Code interactive terminal, tested against the server run from a checkout; this npm route and the plugin have passed the clean-install test suite but not yet a full session in Claude Code. Other MCP hosts are not claimed as verified. Node.js 20 or newer is required.

--input-root and --output-root are required and repeatable. Paths outside them are refused, reports go into new folders only, and existing files are never overwritten. --help and --version print and exit.

Tools

load_predictions, assess_predictions, fit_temperature, set_policy, review_test and create_report. Test outcomes stay unopened until review_test receives an explicit decision through the client's elicitation form. Input formats, the complete workflow, disclosure rules and limits are documented in docs/mcp.md.

What the package contains

dist/cli.js is a readable, unminified bundle of the server, the shared app modules and @m-sanchez/calibrated from the exact Git commit the app pins. The official MCP TypeScript SDK and zod are exact-pinned runtime dependencies. The UCI digits reference ships under public/examples with its CC BY 4.0 attribution. Reports record the same app source and numerical distribution hashes as a checkout of the same commit; dist/NOTICE.txt lists them with the bundled licence.

MIT licence. Copyright (c) 2026 Miguel Sánchez Durán.

Source: packages/mcp/README.md at commit 1d2c32f

Tools

0
Tool metadata has not been indexed yet.

Version history

1
  1. v0.2.0LatestOct 5, 2026