Llmtest Mcp

io.github.tjacquessonPublished by GitHub account tjacquessonv0.6.1Updated Oct 24, 2026

Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.

VerifiedSTDIODesktop onlyData & AnalyticsAI & ML

Overview

AI-generated overview

LLMTest MCP server that lets an assistant benchmark AI models on your real prompts and find cheaper, faster alternatives across 340+ models.

What you can do
Check proxy status, account credit balance and usage, and autopilot eligibility and state, and turn autopilot on or off with its safety gates and revert window. List AI flows with cost and latency stats, and review new and trending models. Get pending model-switch suggestions, accept or dismiss them, and roll back auto-accepted optimizations within their revert window. Run benchmarks of a flow against challenger models, and rewrite a flow's prompt to find a cheaper model that still works. Add and list test prompts for pre-launch benchmarking.
When to use it
Useful when you route AI calls through the LLMTest proxy and want an assistant to review cost and latency, compare models on your own prompts, and act on model-switch suggestions or autopilot optimizations without leaving the chat.
Requirements
Runs locally via npx (Node.js). Requires an LLMTest account and an API key supplied in the LLMTEST_API_KEY environment variable; an optional LLMTEST_BASE_URL overrides the default API URL. Network access to LLMTest is needed.
Before you install
The API key is a secret passed by environment variable. Several tools change account state: accepting or dismissing suggestions, enabling or disabling autopilot, reverting optimizations, and rewriting prompts. Autopilot can change live prompts automatically and consumes credits (roughly $1–2 per run), so a positive balance is required. Your prompts and usage data are sent to LLMTest.

Installation

In SourceWeft

  1. Open Llmtest Mcp in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

LLMTest MCP Server

[npm version] [MIT License]

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

Sign up at llmtest.io and grab your API key from the dashboard.

2. Add to your tool

Claude Code:

bash
claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

bash
export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

json
{  "mcpServers": {    "llmtest": {      "command": "npx",      "args": ["llmtest-mcp"],      "env": {        "LLMTEST_API_KEY": "llmt_your_key_here"      }    }  }}

3. Talk to your AI

Just ask in natural language:

  • "Check my LLMTest status"
  • "Find cheaper models for my AI calls"
  • "Run a benchmark on my blog-writer flow"
  • "What models are trending?"

How It Works

LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.

This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.

Available Tools

ToolDescription
statusShow proxy status and activity summary
list_flowsList all AI flows with cost and latency stats
get_suggestionsGet pending model-switch recommendations
update_suggestionAccept or dismiss a suggestion
run_benchmarkBenchmark a flow against challenger models
optimize_promptRewrite a flow's prompt and find a cheaper model that still works
seed_samplesAdd test prompts for pre-launch benchmarking
list_samplesShow stored test samples per flow
list_new_modelsShow new and trending models
get_accountCheck credit balance and usage
get_autopilot_statusCheck whether autopilot is on and whether the account is eligible
enable_autopilotTurn on weekly auto-optimization with safety gates + drift-based auto-revert
disable_autopilotTurn off autopilot (existing optimizations stay active)
list_active_optimizationsList auto-accepted optimizations still inside their 24h revert window
revert_optimizationRoll an auto-accepted optimization back to the previous prompt

Autopilot

Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.

To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enable_autopilot. Use get_autopilot_status to confirm prerequisites.

Prerequisites (checked per flow each cycle):

  • Autopilot enabled on the account
  • Email verified
  • Account age ≥ 14 days (trust ramp)
  • Flow has ≥ 20 real calls in the last 7 days
  • Flow not optimized by autopilot in the last 14 days (cooldown)
  • Positive credit balance (~$1–2 per run)

Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.

Revert: 24h window after auto-accept. After that, only drift detection can roll back.

Typical Workflow

Pre-launch (no traffic yet):

  1. Tell your AI: "I'm building a support chatbot using gpt-4o"
  2. It seeds realistic test samples with seed_samples
  3. It runs run_benchmark to compare models
  4. It shows you get_suggestions with cheaper alternatives

Post-launch (with real traffic):

  1. Route your AI calls through https://llmtest.io/v1
  2. LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls
  3. Ask "any cost-saving suggestions?" to see recommendations
  4. Accept a suggestion and update your code

Environment Variables

VariableRequiredDescription
LLMTEST_API_KEYYesYour API key from llmtest.io/dashboard
LLMTEST_BASE_URLNoCustom API URL (defaults to https://llmtest.io)

Links

License

MIT

Source: README.md at commit d00dcff

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.6.1LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts