Modelwatch

io.github.bch1212Published by GitHub account bch1212v0.1.3Updated Oct 24, 2026

Continuous behavioral drift monitoring for LLM apps — catches silent provider model updates.

VerifiedSTDIODesktop onlySecurity & MonitoringAI & ML

Overview

AI-generated overview

ModelWatch MCP lets an assistant create and run behavioral drift specs for LLM endpoints, then report drift events and per-spec health.

What you can do
List and add monitored LLM endpoints by provider and model, and list or create behavioral specs with a prompt, schedule and optional similarity threshold. Run a spec on demand, establishing a baseline on the first run, and reset the baseline after an intentional model swap. Review recent drift events across the workspace and the run history for a single spec. Check per-spec health, including last drift score and last run time.
When to use it
Useful when you build on hosted LLM providers and want to catch silent model updates that change behavior, for example refusal canaries or prompt regression checks. It suits teams that want to set up and inspect drift monitoring from inside a chat client rather than a separate dashboard.
Requirements
Runs as a local stdio process, installed from the npm package modelwatch-mcp (Node.js required). Needs a ModelWatch API key in MODELWATCH_API_KEY; the free tier allows 5 specs and 500 runs per month. MODELWATCH_API_BASE optionally overrides the API base URL for self-hosted deployments, which must be an HTTPS origin except for loopback development. Network access to the ModelWatch API is required.
Before you install
The API key is a secret stored in plaintext in the client config file; restrict that file, do not commit it, and rotate the key if exposed. The README states the key should never be passed as a command-line argument. Creating endpoints and specs and running specs writes data to your ModelWatch workspace, and specs can run on a schedule and email you when drift is detected. Prompts and results are sent to the ModelWatch service.

Installation

In SourceWeft

  1. Open Modelwatch in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

modelwatch-mcp

MCP server for ModelWatch — continuous behavioral drift monitoring for LLM-powered applications.

Lets Claude Desktop / Claude Code (or any MCP-compatible client) create drift specs, run them, and surface drift events without leaving the chat.

Install

bash
npm install -g modelwatch-mcp

Configure (Claude Desktop)

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

json
{  "mcpServers": {    "modelwatch": {      "command": "npx",      "args": ["-y", "modelwatch-mcp"],      "env": {        "MODELWATCH_API_KEY": "mw_PASTE_FROM_EMAIL"      }    }  }}

Get a free key at https://modelwatch.app/#signup (5 specs, 500 runs/mo, no card). Then restart Claude Desktop.

The example configuration stores a plaintext key in a local config file. Restrict that file to your user account, do not commit it, and rotate the key if it is exposed. The key never belongs in a command-line argument.

Tools

ToolWhat it does
list_endpointsShow monitored LLM endpoints
create_endpointAdd an endpoint (provider + model)
list_specsShow behavioral specs
create_specAdd a spec (prompt + schedule + optional semantic similarity threshold)
run_specRun a spec on demand; baseline if first run
reset_baselineRe-baseline after an intentional model swap
get_drift_eventsRecent drift events across the workspace
get_spec_historyRun history for one spec
get_healthPer-spec health status, last drift score and last run time

Example session

You: Create a refusal canary against the OpenAI endpoint, asking about chest pain. Run it now and tell me the baseline.

Claude calls list_endpoints → create_spec → run_spec and reports the score. The same spec then runs daily forever and emails you when drift is detected.

Environment variables

VarRequiredDefault
MODELWATCH_API_KEYYes—
MODELWATCH_API_BASENohttps://api.modelwatch.app

Self-hosted? Point MODELWATCH_API_BASE at an HTTPS origin. HTTP is accepted only for loopback development; paths, query strings, URL credentials and redirects are rejected before a key is sent.

Source

https://github.com/bch1212/modelwatch — MIT licensed.

Source: mcp-server/README.md at commit 76d6acb

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

3
  1. v0.1.3LatestOct 7, 2026
  2. v0.1.2Oct 1, 2026
  3. v0.1.1Sep 16, 2026

Score

40% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts

    Declare a license in the repository or the registry entry.