LEEVAR reliability battery

io.github.forevercrab321-svgPublished by GitHub account forevercrab321-svgv0.1.0Updated Oct 24, 2026

Grade an AI agent's transcripts on 18 reliability tests. Thin evidence is NOT TESTED, not guessed.

VerifiedSTDIODesktop onlyData & AnalyticsAI & ML

Overview

AI-generated overview

LEEVAR reliability battery grades an AI agent's conversation transcripts against 18 reliability tests and returns a coverage-aware report.

What you can do
Send 1–8 real conversation samples to the hosted battery and get a scan id back, with a free allowance of 5 scans a month per email and no account needed. Read a scan's status, grade (only when one was issued), coverage and written report. List the 18 tests, what each checks and the failure it catches, to help pick samples that exercise them. Dimensions the samples cannot evidence come back as NOT TESTED, and no grade is issued below 13 of 18 evidenced tests.
When to use it
Useful when you want an outside check on how a support bot or other AI agent behaves across real conversations, and want to see which reliability behaviours the samples actually cover before trusting a letter grade.
Requirements
Runs as a local stdio process on the user's machine; Node 18 or later. The package is installed from the v0.1.0 release rather than the npm registry. Network access to LEEVAR's hosted service is required, and each scan draws on the email's 5 free monthly scans.
Before you install
Conversation samples are sent to LEEVAR's hosted service and graded there by third-party language models; they are deleted 30 days after the scan completes. Strip API keys, credentials and personal data before sending. The tool descriptions tell the assistant to ask before sending anything. A reference average shown without a grade is not a verdict.

Installation

In SourceWeft

  1. Open LEEVAR reliability battery in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

leevar-mcp

The LEEVAR reliability battery as an MCP server. Ask Claude, Cursor or any MCP client to grade an AI agent's conversations on 18 reliability tests, without leaving the editor.

It does not grade anything itself and it never fills a gap with a guess. A dimension your samples cannot evidence comes back NOT TESTED, and below 13 of 18 evidenced tests no grade is issued at all — the tool says so instead of printing a letter.

Tools

ToolWhat it does
leevar_scan_transcriptSends 1–8 real conversation samples to the hosted battery. Returns a scan id. Free: 5 scans a month per email, no account.
leevar_scan_statusReads the result: status, grade (only when one was issued), coverage, and the written report.
leevar_batteryLists the 18 tests — what each checks and the failure it catches — so you can pick samples that exercise them.

Install

Node 18 or later. The package is installed straight from the v0.1.0 release; it is not on the npm registry yet.

Claude Desktop (claude_desktop_config.json) or Cursor (.cursor/mcp.json):

json
{  "mcpServers": {    "leevar": {      "command": "npx",      "args": ["-y", "https://github.com/forevercrab321-svg/leevar-battery/releases/download/v0.1.0/leevar-mcp-0.1.0.tgz"]    }  }}

Claude Code:

bash
claude mcp add leevar -- npx -y https://github.com/forevercrab321-svg/leevar-battery/releases/download/v0.1.0/leevar-mcp-0.1.0.tgz

From source instead: clone this repository, run npm install in packages/mcp, and point your client at node <clone>/packages/mcp/index.mjs.

Then ask, for example: "List the LEEVAR battery, pick three of my support bot's conversations that cover it, and scan them."

Your data

Samples go to LEEVAR's hosted service, are graded there by third-party language models (named on the privacy page), and are deleted 30 days after the scan completes. Each scan uses one of the email's 5 free monthly scans. Strip API keys, credentials and personal data before sending. The tool descriptions tell the assistant to ask you before it sends anything.

To grade entirely on your own machine with a model you choose, use the scanner in the root of this repository instead (deno task scan) — nothing leaves your machine except the calls to your own model provider.

Reading a result

  • Grade issued — Grade: B (composite 81.2) with the coverage it was computed on. Gate on coverage as well as the letter.
  • No grade — the samples did not evidence enough of the battery. Send longer, real conversations that exercise the missing behaviours: a long thread for context, a repeated task for consistency, a tool failure for recovery. A reference average may be shown; it is not a verdict.
  • Failed — the reason, whose fault it was, and whether a retry makes sense. If the fault is LEEVAR's, the free scan is credited back.

Develop

bash
npm installnpm test          # all requests go to local fakes; no free scans are spentnpm run gen       # regenerate battery.json from packages/scanner-core (Deno)

LEEVAR_API_URL overrides the endpoint (the tests use it to point at a local stub).

MIT.

Source: packages/mcp/README.md at commit af11f33

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.1.0LatestSep 28, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts