Llmtest Mcp

io.github.tjacquesson由 GitHub 账号 tjacquesson 发布v0.6.1更新于 Oct 24, 2026

Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.

已验证STDIO仅桌面数据与分析AI 与机器学习

概览

AI 生成的概览

LLMTest MCP 服务器，让助手用你的真实提示词对 AI 模型做基准测试，并在 340 多个模型中寻找更便宜、更快的替代方案。

可以做什么
查看代理状态、账户余额与用量，以及 autopilot 的资格与状态，并开启或关闭带安全门槛和回滚窗口的 autopilot。列出各 AI 流程的成本与延迟统计，查看新增和热门模型。获取待处理的模型切换建议，接受或忽略建议，并在回滚窗口内撤销自动接受的优化。针对挑战模型对某个流程运行基准测试，并改写流程提示词以找到仍然可用的更便宜模型。为上线前基准测试添加和列出测试提示词。
适用场景
当你通过 LLMTest 代理转发 AI 调用，并希望助手在不离开对话的情况下查看成本与延迟、用你自己的提示词比较模型，并处理模型切换建议或 autopilot 优化时适用。
运行要求
通过 npx 在本地运行（需要 Node.js）。需要 LLMTest 账户，并通过 LLMTEST_API_KEY 环境变量提供 API 密钥；可选 LLMTEST_BASE_URL 覆盖默认 API 地址。需要能访问 LLMTest 的网络。
安装前请注意
API 密钥是敏感信息，通过环境变量传入。多个工具会改变账户状态：接受或忽略建议、开启或关闭 autopilot、撤销优化以及改写提示词。autopilot 可能自动修改线上提示词并消耗额度（每次运行约 1–2 美元），因此需要正余额。你的提示词和用量数据会发送给 LLMTest。

安装

在 SourceWeft 中

  1. 打开 控制台中的 Llmtest Mcp，将其添加到工作区。
  2. 为需要使用其工具的对话启用该服务。

Desktop only，通过 STDIO。 STDIO 服务会启动本地进程，因此需要 SourceWeft 桌面宿主。

其他 MCP 客户端

参照 仓库 中的启动说明。

README

LLMTest MCP Server

[npm version] [MIT License]

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

Sign up at llmtest.io and grab your API key from the dashboard.

2. Add to your tool

Claude Code:

bash
claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

bash
export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

json
{  "mcpServers": {    "llmtest": {      "command": "npx",      "args": ["llmtest-mcp"],      "env": {        "LLMTEST_API_KEY": "llmt_your_key_here"      }    }  }}

3. Talk to your AI

Just ask in natural language:

  • "Check my LLMTest status"
  • "Find cheaper models for my AI calls"
  • "Run a benchmark on my blog-writer flow"
  • "What models are trending?"

How It Works

LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.

This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.

Available Tools

ToolDescription
statusShow proxy status and activity summary
list_flowsList all AI flows with cost and latency stats
get_suggestionsGet pending model-switch recommendations
update_suggestionAccept or dismiss a suggestion
run_benchmarkBenchmark a flow against challenger models
optimize_promptRewrite a flow's prompt and find a cheaper model that still works
seed_samplesAdd test prompts for pre-launch benchmarking
list_samplesShow stored test samples per flow
list_new_modelsShow new and trending models
get_accountCheck credit balance and usage
get_autopilot_statusCheck whether autopilot is on and whether the account is eligible
enable_autopilotTurn on weekly auto-optimization with safety gates + drift-based auto-revert
disable_autopilotTurn off autopilot (existing optimizations stay active)
list_active_optimizationsList auto-accepted optimizations still inside their 24h revert window
revert_optimizationRoll an auto-accepted optimization back to the previous prompt

Autopilot

Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.

To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enable_autopilot. Use get_autopilot_status to confirm prerequisites.

Prerequisites (checked per flow each cycle):

  • Autopilot enabled on the account
  • Email verified
  • Account age ≥ 14 days (trust ramp)
  • Flow has ≥ 20 real calls in the last 7 days
  • Flow not optimized by autopilot in the last 14 days (cooldown)
  • Positive credit balance (~$1–2 per run)

Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.

Revert: 24h window after auto-accept. After that, only drift detection can roll back.

Typical Workflow

Pre-launch (no traffic yet):

  1. Tell your AI: "I'm building a support chatbot using gpt-4o"
  2. It seeds realistic test samples with seed_samples
  3. It runs run_benchmark to compare models
  4. It shows you get_suggestions with cheaper alternatives

Post-launch (with real traffic):

  1. Route your AI calls through https://llmtest.io/v1
  2. LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls
  3. Ask "any cost-saving suggestions?" to see recommendations
  4. Accept a suggestion and update your code

Environment Variables

VariableRequiredDescription
LLMTEST_API_KEYYesYour API key from llmtest.io/dashboard
LLMTEST_BASE_URLNoCustom API URL (defaults to https://llmtest.io)

Links

License

MIT

来源：README.md，提交 d00dcff

工具

0
SourceWeft 尚未在沙箱中启动过该软件包，因此其工具未知。

版本历史

1
  1. v0.6.1最新Sep 16, 2026

评分

已满足适用检查项的 50%

该评分衡量此收录条目的完整程度，依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价，也不表示服务器安全或当前可用：可连通性是带日期的检查结果，评分也不包含安装量或用户评价。

  • 近 30 天内可连通25 分

    请保持端点能够响应 MCP 握手；SourceWeft 每 30 天探测一次。

  • 已声明认证方式10 分

    请在注册中心条目中声明凭据请求头，或保持端点能够响应探测，以便判定其认证方式。

  • 已列出工具15 分

    请让服务器在无凭据时响应 tools/list，或发布可在沙箱中无凭据启动的软件包。

  • 描述必需15 分
  • 图标10 分
  • 文档15 分
  • 许可证10 分