[toolmenu logo: a plug with a list of tools, one checked]
toolmenu
Checks for MCP servers' tool lists: changes that confuse agents, break prompt caches or break clients, caught before your users hit them.
[npm]
[ci]
[license: MIT]
[MCP: 2026-07-28 and 2025]
[toolmenu]
[toolmenu MCP server on Glama]
Your MCP server's tool list is the menu every agent reads before it does anything.
toolmenu checks that menu the way clients and agents will meet it: across calls,
connections, clients and releases. No LLM is involved, so the same inputs always give
the same answer and it fits in CI.
[toolmenu session output: tools inserted mid-list, a description edited, each pinned to the step that caused it]
What it catches
Every finding says what's wrong, where it is and what toolmenu saw, and ends with a → Next: step.
It runs on the 22-server bench and on hosted servers (GitHub, Atlassian, Sentry,
Notion, Supabase, Vercel…). What it has found on real servers is in
docs/FINDINGS.md.
Status: 0.25. Spec in docs/SPEC.md,
every rule in docs/RULES.md, changes in CHANGELOG.md.
Why "when it changes" matters
The MCP linters I found grade one snapshot of the menu: tool names, descriptions,
schemas. toolmenu also checks when the menu changes:
- Mid-session. Most clients put the tool list at the start of the prompt, so any
change to it, even a tool added at the end, means everything after it is processed
again. More on this, with a benchmark of the setups, in
this write-up.
session pins each
change to the step that caused it, and says where in the list it happened.
- Per connection. 2026-07-28 says the tool set must not vary per connection or
as a side effect of other requests (the 2025 versions allow it). Per-session
"unlock" designs, like the one I built, are ruled out.
session tells you whether
yours is one of them. It also checks the menu clients will actually show: a broken
x-mcp-header or a $ref they can't resolve hides a tool or fails every call to
it. When a tool stops to ask the user something, session answers "no" and says
whether the tool can take it.
- Between releases.
diff finds breaking changes and token growth, and checks the
version bump. history does it across a package's published releases. It installs
old versions with today's dependencies, so it records what they serve now
(FINDINGS F4).
Run against the reference server, @modelcontextprotocol/server-everything, it
reports no false findings (bench/).
Start here
Needs Node 22 or later. The GitHub Action brings its own if the runner's Node is older,
for toolmenu only: your server and the job's later steps keep the job's Node.
Your server can be in any language: toolmenu starts it (stdio) or connects to it (HTTP).
One command sets up CI: it snapshots your server into menu.json (the baseline) and
writes .github/workflows/toolmenu.yml for your project (Node, Python, Go or Rust).
Commit both, open a pull request, and toolmenu comments on it.
Every value you pass with --env or --header goes into the workflow as
${{ secrets.… }}, never as text (a name like DATABASE_URL doesn't say it holds a
password); init says which secrets to add. It never overwrites a file.
Run it anywhere in the repository. .github/workflows/toolmenu.yml goes to the git
root, and the server starts in CI from the folder you ran init in, which is also
where menu.json is saved. That's how a monorepo package works: run init in
packages/server. Paths to files in the repository are made relative, so they work on
the runner. init lists anything that won't work there (a file outside the
repository, a localhost URL, a virtualenv the runner doesn't have), each with what
to do.
By hand:
HTTP servers work the same way: npx toolmenu snapshot https://example.com/mcp --header "Authorization: Bearer $TOKEN".
Servers behind OAuth (most hosted ones: GitHub, Atlassian, Sentry…): log in once, in a
browser, and snapshot and session use that login from then on, refreshing it as
needed:
Logins are stored per server in ~/.config/toolmenu/auth (readable only by you);
auth list and auth logout <url> manage them. A server that doesn't allow
dynamic registration needs a pre-registered app: --client-id (and
--client-secret). Over SSH or in a container, auth login --no-browser prints the
login URL instead of opening it: open it on any device, sign in, and paste back the
address the browser ends on (a localhost page that doesn't load is expected). In
CI, where nobody can sign in, pass a token as a header.
Both protocol generations are supported: 2026-07-28 (server/discover, stateless)
and the 2025 initialize handshake, through the official TypeScript SDK.
Every finding and error says what's wrong, where, what toolmenu saw, and ends in a
→ Next: step. Findings marked · unsure are heuristics (names and words): they say
what was seen, not what it means. toolmenu explain <rule> says what a rule checks,
its default level and how to change it (every rule is in docs/RULES.md).
Every command has its own help: toolmenu history --help.
Behind a proxy (HTTPS_PROXY), stdio servers get the proxy and CA settings
(HTTPS_PROXY, NO_PROXY, NODE_EXTRA_CA_CERTS, SSL_CERT_FILE, REQUESTS_CA_BUNDLE…)
along with the minimal environment, and NODE_USE_ENV_PROXY=1 so a Node server's
fetch uses the proxy. Without them a server that calls an API can't reach it, and
fails as if it were broken.
Every server you use: scan
scan runs snapshot on every server in an MCP client config (Claude Desktop, Claude
Code's .mcp.json, Cursor, VS Code: the mcpServers or servers object) or in a
text file with one server per line (a URL or a command, optionally name<TAB> first),
four at a time (--concurrency), and prints them side by side, then each server's
findings, like this:
${VAR}, ${VAR:-default} and ${env:VAR} in the config are filled from the
environment. A server whose variable isn't set isn't started with an empty
credential: it's a row that says so. A server that couldn't be started or read
makes the exit code 2, because it wasn't checked; one marked disabled is listed and
left alone. URLs are printed without their query, which can hold a key. Nothing is
written unless --out-dir names a folder for each server's menu file.
For the agent building your server: mcp
toolmenu mcp is toolmenu as an MCP server (stdio), so Claude Code, Cursor or any
other client can check the server it's working on while it works. In Claude Code, as a
plugin (the server, and skills for checking a change, checking a release, and setting up CI):
Or just the server, in any client: claude mcp add toolmenu -- npx -y toolmenu mcp, or
{"command": "npx", "args": ["-y", "toolmenu", "mcp"]} in an mcpServers config.
Each tool runs the CLI in a child process and returns its text report, so options,
toolmenu.config.json and stored OAuth logins work as they do on the command line.
Relative paths resolve from the folder toolmenu mcp runs in, or a call's cwd.
snapshot_menu and check_session start the command you give them, as the CLI does;
clients ask before each tool call unless you allow it. toolmenu's own menu passes its
checks with no findings, on both protocols and in a session (the test suite checks).
In CI: one PR comment
It snapshots the PR's build, diffs it against the baseline as it is on the PR's base
branch (or at the previous tag, on a tag push), runs the scenario, and
posts one comment that updates on every push: breaking changes, the token change
("this PR adds ~1,240 tokens to every conversation"), the suggested version bump,
and anything session caught. The same report goes to the job summary.
In the Security tab: code scanning
With sarif, findings also go to GitHub code scanning: alerts in the Security tab,
each on the line of its tool in your menu.json, and a code scanning check on the pull
request. GitHub shows an alert inline in the pull request only when that line changed in
it, so most stay in the Security tab and the check. Run the workflow on pushes to your
default branch too, so GitHub has a base to compare a pull request's alerts with. Code scanning is free for public repositories; private ones need GitHub Advanced
Security.
Outside the Action: toolmenu snapshot --sarif toolmenu.sarif -- node dist/server.js
writes the file beside the usual report (diff and session take --sarif too), or
--format sarif prints it instead.
A badge for your README
writes a shields.io endpoint file (tools,
tokens, and errors or warnings, in red, yellow or green) and prints the README line
for it. Commit it with menu.json: it changes when the baseline does, so nothing
has to commit from CI. The badge at the top of this README is toolmenu's own MCP
server (toolmenu mcp), checked by itself.
Server behind auth, or PRs from forks and Dependabot? See docs/github-action.md.
Server that unlocks tools on request? Snapshot the full menu for the baseline and run the session on the default one: servers that unlock tools.
PRs without secrets are skipped, and a skipped check still passes; the skipped output tells you.
session: the menu changing while the agent works
session runs a scripted session (the scenario is the agent), lists the menu
after every step, and reports each change with the step that caused it.
--init writes a starter scenario from the live menu: it calls every read-only tool
that needs no arguments, then repeats one. If a tool looks like it unlocks more tools
(a domains or toolset parameter with an enum, "unlock" or "capabilities" in its
name, a description that says it enables or loads tools), the starter unlocks
every value, listing after each, then repeats the first, so the session sees every
tool behind the unlock. The values come from the parameter's enum or, without one,
from the server's own read-only listing (list_toolsets). Lookups that only describe
toolsets (list_toolsets, get_toolset_tools) aren't unlocks, and one not marked
read-only is suggested, never called. Anything it can't fill in is left as a
commented-out step. An unlock nothing in the menu names, like joining a room that
brings more tools, needs a step you add yourself. A scenario is only as good as its
steps, so the starter is a floor, not a ceiling.
On github-mcp-server 1.0.5 with --dynamic-toolsets (3 tools at the start, 19
toolsets), unlocking every toolset reaches 107 tools: all 81 that --toolsets all
serves, and 23 issue and pull-request tools that only the dynamic mode has.
No scenario? session --auto builds the steps from the menu: every read-only tool
whose required arguments the schema itself gives (a default, examples, an enum,
a type or format), cheapest first, then every value of each read-only unlock (outside
--max-calls, and the first value twice: a second identical unlock should change
nothing), then the first call again. It never guesses an ID: a tool that needs
owner, repo or an issue key is skipped, and the report names the flag that fills
it: --value repo_path=/src/app fills that parameter on every tool that requires it,
--value get_issue.issue_key=ABC-1 on one tool, --values-file values.yml takes a map
of either. Servers that don't mark their tools readOnlyHint (DeepWiki, every
2024-11-05 server) get nothing called unless you vouch for them by name:
--assume-read-only read_wiki_structure,ask_wiki_question. Only exact names are
accepted: a pattern would also cover tools a later release adds. A tool marked or
named as a write is never called, and the report says which tools ran on your word.
With --create-handles on 2026-07-28, --auto also calls the tool that creates a
session, basket or cart (start_session; a write, so only with the flag, and never
one marked destructiveHint; not locks, leases or checkouts) and gives the tools that
take it back the handle it returned, beside the made-up one.
--save-scenario auto.yml writes the steps, with the skipped tools commented out, for
you to fill in. Tools marked openWorldHint: true (web
search, fetch, scraping, which can cost API credits) are called only with
--open-world, except unlocks: changing the menu is what a session watches, and an
unlock spends no search credits, so it runs even on a server that marks every tool
open-world. A run that ends up calling nothing is session/nothing-called (warn),
with why each tool was left out, not a clean pass. On the 22 servers in bench/,
--auto reaches 60 of 262 read-only tools, 84 with --open-world.
session calls tools for real. Without allow_writes: true it refuses any tool
not marked readOnlyHint: true. --plan prints the steps without connecting.
diff: compare releases
That's two real releases of the official filesystem server. Its calendar versions
promise nothing about compatibility, so the bump isn't judged, but the breaking change
is still an error. Changes are classified
the way OpenAPI breaking-change checkers do it:
The same rules apply inside parameters: fields of an object parameter, array items,
and the options of an anyOf/oneOf union (zod's unions, discriminated unions and
.nullable()) are compared at every depth, with their path (gen.body.text,
search.filters[].field, gen.block(kind="image").url). Union options are paired by
the value of a discriminator they all fix; without one (a plain z.union of objects),
identical options first, so a reorder is no change, then by type and shared property
names. An option gone is breaking, a new one widens. The same change reached through
one shared definition at several places is one finding that lists them (places in
--json). Local $refs (#/$defs/…, #/definitions/…) are expanded first,
recursive ones one level deep, so a schema is compared by what it accepts, not how
it's spelled: moving a repeated block into $defs is one diff/schema-equivalent
notice with its token change ("accepts the same input: ~899 tokens fewer"), and a
breaking change inside $defs is breaking. A schema change no rule classifies is
never silent: it's a diff/schema-other "review it".
On mongodb-mcp-server 2.1.2 → 3.0.0 this finds aggregate.pipeline[]…$vectorSearch.numCandidates
and .limit narrowed from number to integer inside a union, where the whole
pipeline used to be one "review it".
Different spellings of the same schema are one schema (additionalProperties absent,
true or {}; zod 4's safe-integer bounds), and a change every tool shares is said
once for the menu: a server moving from zod 3 to zod 4 reads as its real changes (a
field that became required, the patterns zod 4 adds to email and uuid) plus one
diff/properties-opened line ("27 tools now accept properties they don't list … most
likely a schema generator upgrade"), not a "review it" per object.
The same change in many tools is reported once, with the tools it touches (pageId
is new and required in 25 of 29 tools, on chrome-devtools-mcp 1.8.0); counts are per
change. When a tool loses a parameter and gains a required one with a close name,
diff/param-renamed (warn, unsure) points out the likely rename; both changes stay
breaking. Every breaking finding's next step names the version to release when the
bump is too small.
It also reports the token change per tool and suggests a semver bump. To check the
bump, pass the release versions: --release 1.4.0..1.5.0 (npm, a git tag).
diff/version-bump then warns when the release's bump is smaller. Under 1.0.0 it
reads versions as npm's caret does (^0.2.3 accepts any 0.2.x): breaking changes need
a minor bump, new features a patch, and under 0.1.0 anything goes.
diff/version-backwards warns when the version goes down. Calendar versions and prereleases aren't judged. The version in the snapshot is what the
server reports (serverInfo.version), which is often not the release: the filesystem
server has said 0.2.0 for 19 releases (FINDINGS F5). It's shown, and only checked
with --server-version-is-release. diff also enforces an optional tokenBudget
(diff/token-budget). Order changes are only a notice here: between releases, prompt caches rebuild. Order
matters within a session.
Operations behind a search tool (--catalog)
Big servers increasingly keep most operations out of the menu: a small fixed menu, a
search tool, and an execute tool (Sentry's search_sentry_tools + execute_sentry_tool,
Atlassian's discover + executeRead). A breaking change to one of those operations
never shows in the menu. snapshot --catalog asks the search tool a fixed set of
queries and keeps the operations it returns in menu.json; diff then compares
operations found in both snapshots with the same rules as tools.
A search returns only its top matches, so toolmenu crawls the catalog by its own
words: it starts from the search tool's own example phrases and the nouns in your tool
names, then searches for every operation it finds or sees mentioned, until 20 queries
in a row turn up nothing new (at most 200). On Atlassian's hosted server that finds all
154 operations behind its 21 tools; on Sentry's, 65 behind 9, the same on every run.
Queries are paced and a rate limit is waited out. An operation not found this time is a
notice (diff/catalog-missing: "not returned by the same queries"), never "removed";
one found only now is diff/catalog-added, and catalogs read with different queries,
or partly (a query failed), are diff/catalog-queries. To steer it, set
"catalog": { "queries": [...], "maxQueries": 200, "crawl": true } in
toolmenu.config.json (FINDINGS F15).
A crawl through search is a lower bound, not a proof: an operation that nothing in the
catalog names, and whose words no query uses, stays unfound. On @sentry/mcp-server
0.42 (stdio) it finds 63 of the 64 operations in the package's catalog; whoami comes
back only when asked for by name. Add queries like that to catalog.queries.
Command routers. Some servers put their operations behind router tools instead:
one tool per area that takes a command and its parameters, and lists its commands
when called with learn: true (Azure's MCP server serves 71 such tools in front of
~410 operations). --catalog detects them (a command parameter, an arguments
object, a boolean listing flag, and a description that says "router" or "sub
commands") and calls each once in its listing mode, never with a command, so nothing
runs. Routers whose command is required aren't called. Detection missed one? Name
them: "catalog": { "routers": ["keyvault", "storage"] }. A router lists everything
it has, so unlike a search the catalog is complete except for routers that failed
(catalog/failed says which, and why).
history: a package's releases, researched
It installs each published version into a temp directory (install scripts off),
snapshots it, and diffs it against the previous version that worked. For every
version it records the protocol, the menu, the findings and the dependency
versions that actually got installed: an old version installed today resolves its
dependencies today, and can serve a different menu than it shipped with.
It's best effort: packages that need env vars, arguments or auth are recorded as
failed, with a reason, never skipped. It installs and runs third-party code. Run it
in a container.
Where the tokens go
snapshot also shows what the menu's tokens are spent on: the biggest tools (split
into description and schema), large enums, and waste you can remove:
- Repeated blocks: the same piece of schema at several places, inside one tool (a
$defs entry could hold it once: "one ~230-token block ×5 in
content_generateMessageHtml") or across tools (paid once per tool: mongodb-mcp-server
3.0.4 repeats a ~2,200-token pipeline schema in 3 tools). Only the largest repeated
block is named, not the pieces inside it.
- Unused
$defs: definitions nothing in the tool refers to. Notion's server 2.5.2
ships the same 9 in all 24 tools, 72% of its menu.
Every tool's estimate is in the menu file, for the ones the summary leaves out:
jq -r '.tools[] | "\(.tokens)\t\(.name)"' menu.json | sort -rn.
snapshot rules
Every rule comes from a failure I actually hit building 115 tools for one MCP
server, or from the spec itself. If a rule can't point to one, it doesn't ship.
The full list, diff and session rules included, is docs/RULES.md.
The naming and ID rules are heuristics, tuned on real servers (FINDINGS F7 and
F10) and a private 115-tool one. They're the extras; the change checks are the point.
Pin routing with routes.yml. Two tools fighting over one word needs a regression
test, not a rename you'll forget:
Matching is keyword matching, the way a keyword or BM25 tool search sees the menu: a
tool matches if every word of the route is in its name or description (the exact
phrase scores higher). It can't read "not", and neither can those searches: a
disclaimer like "not the audit trail" puts "audit trail" into exactly the tool you
meant to steer away from, and toolmenu says so.
Config (optional toolmenu.config.json):
offlineCheck turns on --offline-check for every snapshot of a stdio server.
descriptionLimit is where your client cuts tool descriptions: a number, or a client
whose cut has been reported: "claude-code" (the default: Claude Code's own docs,
code.claude.com/docs/en/mcp, read 2026-11-02:
16,384 characters for a tool loaded through tool search, which is on by default, and
4,096 for a tool that loads up front, marked _meta["anthropic/alwaysLoad"]: true;
the cut was 2,048 before v2.1.296), "claude-code-upfront" (4,096 for every tool:
Claude Code with tool search off) or
"amazon-q" (10,024, reported in
makenotion/notion-mcp-server#145).
Clients that send descriptions in full don't need a setting beyond that. If your own
client cuts shorter, as mine did at 280, set its number. fullDescriptions lists the
tools it sends uncut, so the description rules skip them.
Checking for one client (--client)
What an agent sees depends on its client. snapshot --client <name> (or "client" in the
config) checks the menu against what that client's own docs say, and every value
toolmenu uses names its source and the date it was read (src/clients.ts). What a
client doesn't document is left out, not guessed.
The snapshot adds a line such as as Claude Code: up front ~524 tokens · on demand ~344 per tool a search loads (largest: update_contacts, ~2,093), and --json a client
object. With a client that defers tools, search/findable runs at any menu size.
What toolmenu can't catch
toolmenu sees what the server sends. Plenty of agent failures happen elsewhere:
- Routing to a feature with no tools. If the agent reaches for something your
menu doesn't have, no menu check can see the gap.
- What the client does with the menu. Truncating descriptions, sorting or
filtering tools, merging several servers. The description rules use Claude Code's
reported cut unless you set yours (
descriptionLimit); they can't see your client.
- Your code and your logs. A capability check that's answered with one yes/no
for two different behaviours, or a log line that reads the same whether something
happened or not. Those live in the client or the server's code.
- Whether the agent actually picks the right tool. That needs evals with a model.
toolmenu stays deterministic on purpose.
Options
Exit codes: 0 clean · 1 findings at or above --fail-on · 2 couldn't connect
(for scan: a server wasn't checked) or bad usage. toolmenu <command> --help lists that command's own options.
Limitations
- Server order isn't final client order. Clients merge tools from several servers
and may sort or filter them. A stable server menu is necessary for cache hits, not
sufficient. toolmenu measures what the server controls.
- Token counts are estimates (
o200k_base) of what the model reads: each tool's
name, description and input schema. Output schemas, annotations, icons and _meta
stay with the client and aren't counted. Vendors tokenize, cache and bill
differently. diff recounts both menus with the toolmenu that runs it, so a
baseline from an older version still compares fairly; a number quoted elsewhere (a
commit message, a dashboard) should say which toolmenu counted it (0.7.1 changed the
count for menus with icons by up to 5×, FINDINGS F7).
- Heuristic rules say so in their messages.
- Not a security scanner or a full conformance suite. Other tools do those well.
About
Built by Nikša Perović while running a 115-tool MCP server and watching
agents trip over its menu. Every rule started as one of those failures, and the
research behind them is in docs/FINDINGS.md. The story of building it, and where it was wrong, is on niksa.me.
Found a false positive, or a failure toolmenu should catch?
Open an issue with the menu (a
menu.json is enough) and what you expected. Changes are listed in
CHANGELOG.md.
License
MIT