EvalForge Lite

io.github.thejaredchapmanv1.0.1Updated Oct 1, 2026

Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.

VerifiedSTDIODesktop onlyAI & MLData & Analytics

Overview

AI-generated overview

Compares up to four text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry on your own prompts, grading answers automatically.

What it does
EvalForge Lite sends the same test prompts to as many as four models across four backends and scores the answers with a judge model plus rule checks such as contains, regex, json_valid and max_length. It returns a leaderboard with 0-100 scores, letter grades, latency, tokens per second and estimated cost, and can block prompts that violate an uploaded company policy. Nine MCP tools cover listing and suggesting models, checking availability, setting policy, evaluating a prompt, running a comparison, listing runs, and fetching reports as text or CSV.
When to use it
Use it when you want evidence about which model fits your own tasks rather than a generic benchmark, or when you need to compare one model on two platforms in a single run. It also suits quick side-by-side quality, speed and cost checks before committing to a provider.
Requirements
Runs locally over stdio, typically via uvx from the PyPI package evalforge-lite; the web app needs Python 3.10 or newer. Provider credentials are required per backend: an OpenRouter API key, or a region plus Bedrock API key or AWS access keys, or a Vertex project id and region with an access token or service-account JSON, or a Foundry resource name and region with an API key or Entra ID token. The optional OPENROUTER_API_KEY environment variable lets the server hold that key so tool calls need…
Before you install
It handles real provider credentials, including AWS access keys, service-account JSON and Entra ID tokens, and sends your prompts and rubrics to the selected model providers and to a judge backend. Runs cost money on the provider side; Bedrock, Vertex and Foundry figures are catalog estimates, not your cloud bill. State is in memory only and is cleared on restart, and the app does not read .env by itself.

Installation

In SourceWeft

  1. Open EvalForge Lite in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

EvalForge Lite

Compare text LLMs side by side. Write a few test prompts, pick up to four models across OpenRouter, Amazon Bedrock, Google Vertex AI and Microsoft Foundry, and EvalForge Lite sends the same prompts to all of them, scores the answers automatically, and shows a leaderboard with letter grades, response time, speed and estimated cost. Use it as a web app or as an MCP server for Claude and other assistants. You bring your own credentials, or the person hosting it keeps them on the server for you.

Documentation site: https://thejaredchapman.github.io/evalforge-lite/

Why use it

  • Test on your own prompts. Choose models from evidence about your tasks, not a generic benchmark.
  • Up to 4 models per run, across 4 backends. X (OpenRouter) and X@bedrock are separate targets, so you can check one model on two platforms in a single run.
  • Automatic grading. A judge model scores each answer against your rubric, and rule checks (contains, regex, json_valid, max_length, available through the API and MCP) add a pass or fail. You get a 0-100 score and a letter grade.
  • A second opinion on every response. Each answer is also evaluated on six criteria (answered, quality, instruction following, completeness, helpfulness, safety) with strengths and weaknesses written out.
  • Speed and cost beside quality. Latency, tokens per second and estimated cost for every model, and a "What matters most?" selector that moves the "Best for ..." badge without a new run.
  • Policy gate. Upload a company policy and prompts that violate it are blocked before any model is called. If the check itself fails, the prompt is blocked.
  • Reports. Download a PDF or a CSV for any of your last five runs.
  • No accounts, no database. Credentials are used for one request and not stored. Nothing is written to disk.

Quick start

1. Run the web app on your computer

Requires Python 3.10 or newer (3.12 recommended; download from https://www.python.org/downloads/) and git.

git clone https://github.com/thejaredchapman/evalforge-lite.gitcd evalforge-litepython3.12 -m venv venvsource venv/bin/activatepip install -r requirements.txtpython app.py

Open http://localhost:8000, paste a key for at least one backend (an OpenRouter API key is the quickest: https://openrouter.ai/workspaces/default/keys), add a test case, pick two to four models, and click Run comparison. Runs are limited to 3 per 8 hours per browser session. Full walkthrough: Getting started.

2. Use it from Claude (MCP server)

With uv installed:

uvx evalforge-lite

Add it to Claude Code in one line:

claude mcp add evalforge-lite -- uvx evalforge-lite

Or install the Claude Code plugin, which bundles the same server:

claude plugin marketplace add thejaredchapman/evalforge-liteclaude plugin install evalforge-lite@evalforge

Then ask your assistant to compare models. It gets 9 tools: list_models, suggest_models, list_availability, set_policy, evaluate_prompt, run_comparison, list_runs, get_report, get_report_csv. Details, Claude Desktop config and credential shapes: MCP server.

3. Host it for other people

Deploy with the included render.yaml (gunicorn, one worker) or any host that can run gunicorn --workers 1 --threads 4 --bind 0.0.0.0:$PORT app:app. By default every visitor supplies their own key. Optionally keep provider keys on the server with environment variables and a shared daily cap (50 per 24 hours by default). Keep it at one worker: all state is in memory per process. Full guide: Hosting and server-side keys.

Good to know

  • The app does not read .env by itself. To use values from it, run set -a; source .env; set +a before python app.py.
  • Each run is limited to 4 models, and each browser session gets 3 runs per 8 hours.
  • All state lives in memory and is cleared when the server restarts. See Privacy and limits.
  • Upgrading from an older version and reading the CSV or API fields? See the notes in Troubleshooting and FAQ.

Backends and credentials

BackendWhat you provide
OpenRouterOne API key
Amazon BedrockA region, plus a Bedrock API key or AWS access keys (optional session token)
Google Vertex AIA project id and region, plus an access token or service-account JSON
Microsoft FoundryA resource name and region, plus an API key or Entra ID access token

A separate judge backend setting chooses where the judge and policy gate run. Bedrock, Vertex and Foundry costs are estimates from catalog prices, not your cloud bill. See Backends and credentials.

Documentation

PageWhat is in it
OverviewWhat it is, who it is for, the three ways to use it
Getting startedInstall, run, and your first comparison
Web app guideEvery part of the screen, in order
Comparing modelsReading metrics, grades, evaluation, cost and their limits
Backends and credentialsKeys, regions, X@backend targets
MCP serverInstall paths, all 9 tools, example prompts
Hosting and server-side keysDeploying for others, operator-held keys, daily cap
Troubleshooting and FAQCommon messages, fixes, and notes on CSV/API field changes
Privacy and limitsWhat data goes where, what is stored, every limit

Test

pytest tests/ -v

Every model and HTTP call is mocked, so the suite needs no API key and makes no network calls.

Contributing

Contributions are welcome: bug reports, model-catalog updates, new checks, docs, and new backends. See CONTRIBUTING.md for setup, tests and the pull request process. When the app shows an error, the popup's Report an issue on GitHub button opens a pre-filled bug report.

License

MIT

Source: README.md at commit b774029

Tools

0
Tool metadata has not been indexed yet.

Version history

1
  1. v1.0.1LatestOct 1, 2026