Metrillm

io.github.MetriLLMPublished by GitHub account MetriLLMv0.2.6Updated Oct 24, 2026

Benchmark local LLM models — speed, quality & hardware fitness verdict from any MCP client

VerifiedSTDIODesktop onlyData & AnalyticsAI & ML

Overview

AI-generated overview

A local MCP server that benchmarks local LLM models on Ollama or LM Studio, reporting speed, quality and a hardware fitness verdict.

What you can do
List locally available models on a chosen runtime (Ollama or LM Studio). Run a full performance and quality benchmark on a model, or performance only. Retrieve previously stored benchmark results, optionally filtered by model name. Upload a result file to the public MetriLLM leaderboard.
When to use it
Useful when you want an assistant to compare local models on your own hardware, check tokens per second, time to first token and memory use, or decide whether a model is a good fit before relying on it.
Requirements
Runs locally via npx (npm package metrillm-mcp); needs Node.js 20 or newer, a running local runtime (Ollama or LM Studio with its local server enabled), and at least one model available on that runtime. Optional environment variables METRILLM_SUPABASE_URL, METRILLM_SUPABASE_ANON_KEY and METRILLM_PUBLIC_RESULT_BASE_URL override upload defaults.
Before you install
The share_result tool uploads a benchmark result file to a public leaderboard, so results become publicly visible; check the file contents before sharing. Benchmarking loads models locally and consumes compute and memory. Optional upload override variables may carry credentials for self-hosted deployments.

Installation

In SourceWeft

  1. Open Metrillm in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

MetriLLM MCP Server

[npm version]

MCP (Model Context Protocol) server for MetriLLM — benchmark local LLMs directly from Claude Code, Cursor, Windsurf, Continue.dev, or any MCP-compatible client.

Quick Start

Claude Code

bash
claude mcp add metrillm -- npx metrillm-mcp@latest

Claude Desktop

Add to claude_desktop_config.json:

json
{  "mcpServers": {    "metrillm": {      "command": "npx",      "args": ["metrillm-mcp@latest"]    }  }}

Cursor / Windsurf / Continue.dev

Add to your editor's MCP configuration:

json
{  "mcpServers": {    "metrillm": {      "command": "npx",      "args": ["metrillm-mcp@latest"]    }  }}

Prerequisites

  • Node.js >= 20
  • One local runtime available:
  • At least one model available on the selected runtime

Available Tools

list_models

List all locally available LLM models.

ParamTypeDefaultDescription
runtime"ollama" | "lm-studio""ollama"Inference runtime

Example response:

json
{  "models": [    { "name": "llama3.2:3b", "size": 2019393189, "parameterSize": "3.2B", "quantization": "Q4_K_M", "family": "llama" }  ],  "count": 1}

run_benchmark

Run a full benchmark (performance + quality) on a local model.

ParamTypeDefaultDescription
modelstring(required)Model name (e.g. "llama3.2:3b")
runtime"ollama" | "lm-studio""ollama"Inference runtime
perfOnlybooleanfalseIf true, measure performance only (skip quality)

Example response:

json
{  "success": true,  "model": "llama3.2:3b",  "verdict": "GOOD",  "globalScore": 65,  "performance": {    "tokensPerSecond": 42.5,    "ttftMs": 120,    "memoryUsedGB": 2.1,    "memoryPercent": 13  },  "interpretation": "This model runs well on your hardware."}

get_results

Retrieve previous benchmark results stored locally.

ParamTypeDefaultDescription
modelstring(optional)Filter by model name (substring match)
runtime"ollama" | "lm-studio""ollama"Inference runtime

share_result

Upload a result to the public MetriLLM leaderboard.

ParamTypeDescription
resultFilestringAbsolute path to the result JSON file

Uses official MetriLLM upload defaults out of the box.

Optional overrides for self-hosted/staging deployments:

  • METRILLM_SUPABASE_URL
  • METRILLM_SUPABASE_ANON_KEY
  • METRILLM_PUBLIC_RESULT_BASE_URL

Architecture

The MCP server is a thin wrapper around the existing MetriLLM CLI logic:

mcp/src/index.ts  → MCP entry point (stdio transport)mcp/src/tools.ts  → Tool definitions + calls to CLI modules    ↓../src/core/      → CLI logic reused directly../src/commands/  → CLI commands (bench, list)

No code duplication — the MCP server imports CLI modules directly.

Supported Runtimes

RuntimeStatus
OllamaSupported
LM StudioSupported
MLXPlanned
llama.cppPlanned
vLLMPlanned

The runtime parameter is present on every tool to prepare for multi-runtime support. Unimplemented runtimes return a clear error.

License

Apache License 2.0

Source: mcp/README.md at commit 41779e7

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.2.6LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts