Llmtest Mcp

io.github.tjacquesson由 GitHub 帳號 tjacquesson 發布v0.6.1更新於 Oct 24, 2026

Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.

已驗證STDIO僅桌面資料與分析AI 與機器學習

概覽

AI 產生的概覽

LLMTest MCP 伺服器，讓助理用你的真實提示詞對 AI 模型做基準測試，並在 340 多個模型中尋找更便宜、更快的替代方案。

可以做什麼
查看代理狀態、帳戶餘額與用量，以及 autopilot 的資格與狀態，並開啟或關閉具備安全門檻和回復視窗的 autopilot。列出各 AI 流程的成本與延遲統計，查看新增與熱門模型。取得待處理的模型切換建議，接受或略過建議，並在回復視窗內還原自動接受的優化。針對挑戰模型對某個流程執行基準測試，並改寫流程提示詞以找到仍然可用的更便宜模型。為上線前基準測試新增與列出測試提示詞。
適用情境
當你透過 LLMTest 代理轉送 AI 呼叫，並希望助理不必離開對話就能檢視成本與延遲、用你自己的提示詞比較模型，並處理模型切換建議或 autopilot 優化時適用。
執行需求
透過 npx 在本機執行（需要 Node.js）。需要 LLMTest 帳戶，並透過 LLMTEST_API_KEY 環境變數提供 API 金鑰；可選的 LLMTEST_BASE_URL 可覆寫預設 API 位址。需要能連線至 LLMTest 的網路。
安裝前請注意
API 金鑰屬敏感資訊，透過環境變數傳入。多個工具會改變帳戶狀態：接受或略過建議、開啟或關閉 autopilot、還原優化以及改寫提示詞。autopilot 可能自動修改線上提示詞並消耗額度（每次執行約 1–2 美元），因此需要正餘額。你的提示詞與用量資料會傳送給 LLMTest。

安裝

在 SourceWeft 中

  1. 開啟 儀表板中的 Llmtest Mcp，將其新增到工作區。
  2. 為需要使用其工具的對話啟用該服務。

Desktop only，透過 STDIO。 STDIO 服務會啟動本機處理程序，因此需要 SourceWeft 桌面主機。

其他 MCP 客戶端

參照 儲存庫 中的啟動說明。

README

LLMTest MCP Server

[npm version] [MIT License]

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

Sign up at llmtest.io and grab your API key from the dashboard.

2. Add to your tool

Claude Code:

bash
claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

bash
export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

json
{  "mcpServers": {    "llmtest": {      "command": "npx",      "args": ["llmtest-mcp"],      "env": {        "LLMTEST_API_KEY": "llmt_your_key_here"      }    }  }}

3. Talk to your AI

Just ask in natural language:

  • "Check my LLMTest status"
  • "Find cheaper models for my AI calls"
  • "Run a benchmark on my blog-writer flow"
  • "What models are trending?"

How It Works

LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.

This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.

Available Tools

ToolDescription
statusShow proxy status and activity summary
list_flowsList all AI flows with cost and latency stats
get_suggestionsGet pending model-switch recommendations
update_suggestionAccept or dismiss a suggestion
run_benchmarkBenchmark a flow against challenger models
optimize_promptRewrite a flow's prompt and find a cheaper model that still works
seed_samplesAdd test prompts for pre-launch benchmarking
list_samplesShow stored test samples per flow
list_new_modelsShow new and trending models
get_accountCheck credit balance and usage
get_autopilot_statusCheck whether autopilot is on and whether the account is eligible
enable_autopilotTurn on weekly auto-optimization with safety gates + drift-based auto-revert
disable_autopilotTurn off autopilot (existing optimizations stay active)
list_active_optimizationsList auto-accepted optimizations still inside their 24h revert window
revert_optimizationRoll an auto-accepted optimization back to the previous prompt

Autopilot

Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.

To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enable_autopilot. Use get_autopilot_status to confirm prerequisites.

Prerequisites (checked per flow each cycle):

  • Autopilot enabled on the account
  • Email verified
  • Account age ≥ 14 days (trust ramp)
  • Flow has ≥ 20 real calls in the last 7 days
  • Flow not optimized by autopilot in the last 14 days (cooldown)
  • Positive credit balance (~$1–2 per run)

Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.

Revert: 24h window after auto-accept. After that, only drift detection can roll back.

Typical Workflow

Pre-launch (no traffic yet):

  1. Tell your AI: "I'm building a support chatbot using gpt-4o"
  2. It seeds realistic test samples with seed_samples
  3. It runs run_benchmark to compare models
  4. It shows you get_suggestions with cheaper alternatives

Post-launch (with real traffic):

  1. Route your AI calls through https://llmtest.io/v1
  2. LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls
  3. Ask "any cost-saving suggestions?" to see recommendations
  4. Accept a suggestion and update your code

Environment Variables

VariableRequiredDescription
LLMTEST_API_KEYYesYour API key from llmtest.io/dashboard
LLMTEST_BASE_URLNoCustom API URL (defaults to https://llmtest.io)

Links

License

MIT

來源：README.md，提交 d00dcff

工具

0
SourceWeft 尚未在沙箱中啟動過該軟體包，因此其工具未知。

版本歷史

1
  1. v0.6.1最新Sep 16, 2026

評分

已滿足適用檢查項的 50%

該評分衡量此收錄條目的完整程度，依據註冊中心條目、端點與公開來源計算得出。它不是對伺服器本身的評價，也不表示伺服器安全或目前可用：可連通性是帶日期的檢查結果，評分也不包含安裝量或使用者評價。

  • 近 30 天內可連通25 分

    請保持端點能夠回應 MCP 握手；SourceWeft 每 30 天探測一次。

  • 已宣告認證方式10 分

    請在註冊中心條目中宣告憑證請求標頭，或保持端點能夠回應探測，以便判定其認證方式。

  • 已列出工具15 分

    請讓伺服器在無憑證時回應 tools/list，或發布可在沙箱中無憑證啟動的軟體包。

  • 描述必需15 分
  • 圖示10 分
  • 文件15 分
  • 授權條款10 分