conductor

conductor

[CI] [License: MIT] [npm]

Verified tool calls for Claude Code: an MCP server (installable as a plugin) whose 23 tools check their own output in code before you get it.

A model asked to "extract the email" or "redact the phone numbers" is usually right, and nothing tells you when it isn't. conductor puts each job behind a contract: an input schema, an output schema and a mandatory verifier. Deterministic jobs (slugs, base64, URL extraction) run as plain code and never touch a model. Judgment jobs (summaries, PII redaction) run as hermetic claude -p calls, and an answer that fails its verifier is retried with the failure as feedback, handed to a fallback, or returned as a failure; it is never silently passed through.

The skeleton (validate -> execute -> verify -> retry -> fallback -> log) is real code that cannot be skipped; the model is quarantined to the judgment-shaped holes.

[conductor's CLI: slugify returns ok with a verdict score of 1; normalize_phone with 'call me' is rejected by the verifier with feedback explaining the expected shape.]

Quick start

You need Claude Code installed and logged in, and Node.js 20 or newer on your PATH (the plugin starts the server with node).

claude plugin marketplace add fainir/conductorclaude plugin install conductor@conductor

Restart Claude Code (or run /reload-plugins in a running session). Inside a session the same steps are /plugin marketplace add fainir/conductor and /plugin install conductor@conductor. Check that the server is up:

$ claude mcp listplugin:conductor:conductor: node /Users/you/.claude/plugins/cache/conductor/conductor/0.4.1/dist/conductor-mcp.cjs - ✓ Connected

Then ask for the job in plain words, for example "redact the PII in this ticket with conductor". The tools show up as mcp__plugin_conductor_conductor__<tool>.

Update with claude plugin marketplace update conductor && claude plugin update conductor@conductor, remove with claude plugin uninstall conductor@conductor.

Without the plugin system. dist/conductor-mcp.cjs is a committed, self-contained bundle, so a clone needs no install or build step:

git clone https://github.com/fainir/conductorclaude mcp add conductor -s user -- node "$PWD/conductor/dist/conductor-mcp.cjs"

Other MCP clients can run npx -y @fainir/conductor, which serves MCP over stdio. The unrelated conductor-mcp package on npm belongs to someone else.

Usage

Every tool returns the same envelope. Real output from the bundle for base64_encode with {"text": "hello"}:

{ "ok": true, "output": { "encoded": "aGVsbG8=" }, "via": "base64_encode", "attempts": 1, "verdict": { "ok": true, "score": 1 } }

via names the rung that produced the answer (the tool itself, a declared fallback, or freeform for plain Claude), and attempts counts tries on that rung. A failure sets ok: false and the tool result's isError, and the verdict says why, so the calling model can fix its call:

{ "ok": false, "via": "redact_pii", "attempts": 0,  "verdict": { "ok": false, "score": 0, "feedback": "invalid input: input: missing required property 'text'; input: unexpected property 'txt' (allowed: text)" } }

From a clone you can drive the same loop with the CLI (npm ci first). Pure tools answer instantly and cost nothing:

$ npx tsx src/cli.ts slugify '{"text":"Hello, World!"}'{  "ok": true,  "output": {    "slug": "hello-world"  },  "command": "slugify",  "attempts": 1,  "verdict": {    "ok": true,    "score": 1  },  "costUsd": 0}
$ npx tsx src/cli.ts normalize_phone '{"phone":"call me"}'{  "ok": false,  "command": "normalize_phone",  "attempts": 1,  "verdict": {    "ok": false,    "score": 0,    "feedback": "shape: normalized must match /^\\+?\\d{6,15}$/ (an optional leading '+' followed by 6-15 digits)"  },  "costUsd": 0}

The CLI exits 1 when the result fails verification. Model-backed tools work the same way (npx tsx src/cli.ts extract_contact '{"text":"Call Sam at [email protected]"}') and spend a claude -p call per attempt. npx tsx src/cli.ts with no arguments lists the commands, and npx tsx src/cli.ts route "make a url slug" '{"text":"Hello World"}' lets a model pick the command.

Tools

Pure tools (code only, no model call)

Model-backed tools (hermetic claude -p call)

Inputs ending in ? are optional. The first three columns come from the command manifests (npm test fails if they drift); the full JSON Schemas are what the MCP server advertises.

How it works

input -> inputSchema check -> run the command -> verify -> ok? -> result                                 ^                  | no                                 +-- retry with the verifier's feedback (model-backed, up to maxAttempts)                                                    | still no                                     declared fallbacks -> freeform (plain Claude, same schema, same verifier)

• Contracts with mandatory verifiers. Each command is commands/<name>/index.ts: a manifest (input and output JSON Schema, determinism, prompts, fallbacks) plus a verify() export. A command without a verifier cannot be registered (src/library.ts, src/registry.ts). Mechanical checks come first (schema, regex, recompute, grounding); llmJudge in src/verify.ts exists for criteria no code can express.
• Precondition gate. Input that breaks the command's inputSchema is rejected before anything runs, with a message that names the problem.
• Determinism gradient. pure commands are deterministic code: run once, cached, free, and never sent to a model. stochastic commands are run by the model and retried with feedback. effectful is reserved for commands that will need a human gate.
• Fallback to freeform. Every model-backed ladder ends at plain Claude, held to the same output schema and verifier, so the conductor never does worse than a raw call, only better when a verified command applies.
• Hermetic calls. Every model call (src/cc.ts) replaces the system prompt (--system-prompt), loads no settings (--setting-sources ""), enables no tools (--tools ""), saves no session, and constrains the answer with --json-schema. This matters: a naive call once returned the operator's own email instead of the one in the input, because ambient context (global CLAUDE.md, identity, hooks) leaked in.
• Composition. runPipeline (src/pipeline.ts) chains commands; each step is verified before the next runs, so errors cannot silently compound.
• Telemetry and routing. Every attempt is a JSONL line. npm run scores prints per-command reliability from the CLI and eval logs, and route() (src/router.ts) shows those rates to a small model that must pick from an enum of command names.

Configuration

Model-backed calls use whatever account the claude CLI is logged into.

Results (measured by the author)

Evals compare a single naive claude -p call with the conductor on the same inputs:

• Single-shot extraction (eval/run-eval.ts): baseline 100% vs conductor 100%. On easy single-shot tasks a frontier model is already reliable, so there is no gap to close.
• Composition (eval/contact-card.ts, extract_contact -> normalize_phone -> slugify): baseline 80% vs conductor 100%. The baseline emitted a phone number with an invisible Unicode character; the deterministic normalize_phone step is structurally immune to that class of error. Cost: the conductor made more calls, about 2x the baseline.

The reliability gap grows with composition depth and machine-consumed output, not with single easy calls. The evals call claude and cost money; rerun them with REPEATS=2 npx tsx eval/run-eval.ts, REPEATS=2 npx tsx eval/contact-card.ts and REPEATS=1 npx tsx eval/triage.ts (a 4-step pipeline).

Add a command

1. Create commands/<name>/index.ts. A pure command exports manifest, run and verify:

   ts

   import type { CommandManifest, SolveContext, Verdict } from "../../src/types.js";import { check, combine } from "../../src/verify.js";
   export const manifest: CommandManifest = {  name: "reverse_text", // must equal the directory name  description: "Reverse the characters of a string.",  determinism: "pure",  trust: "human-verified",  inputSchema: { type: "object", properties: { text: { type: "string" } }, required: ["text"], additionalProperties: false },  outputSchema: { type: "object", properties: { reversed: { type: "string" } }, required: ["reversed"], additionalProperties: false },  fallbacks: [],  effectful: false,};
   export async function run(input: { text: string }): Promise<{ reversed: string }> {  return { reversed: [...input.text].reverse().join("") };}
   export async function verify(output: { reversed: string }, input: { text: string }, _ctx: SolveContext): Promise<Verdict> {  return combine(    check("string", typeof output?.reversed === "string", "reversed must be a string"),    check("round-trip", [...(output?.reversed ?? "")].reverse().join("") === input.text, "reversing the output must give the input back"),  );}

   A model-backed command sets determinism: "stochastic", leaves out run, and adds systemPrompt, promptTemplate, model, maxAttempts and fallbacks: ["freeform"]; its verify checks the model's answer (see commands/extract_contact/index.ts).

2. Add commands/<name>/tests/cases.json, a list of { "input": ..., "expect"?: ... }.

3. Import the module in src/library.ts and add it to MODULES (the bundle cannot scan directories).

4. Add a row to the right table above. npm test checks contracts, runs pure commands against their cases, and prints the exact README row it expects.

5. npm run build and commit dist/conductor-mcp.cjs, which is what the plugin runs.

Development

npm cinpm test            # hermetic: a fake `claude` on PATH, no model calls, no networknpm run typechecknpm run build       # rebuild dist/conductor-mcp.cjs and commit it (CI fails if it is stale)npm run mcp         # run the server from source over stdionpm run scores      # reliability scoreboard from projects/*/telemetry.jsonl
# These call the real claude CLI and cost money:npx tsx eval/run-tests.ts [command]   # each command's tests/cases.json through the full loopnpm run eval                          # eval/run-eval.ts

npm run release [patch|minor|major] bumps the version in package.json and both plugin manifests, rebuilds, commits and pushes; installed users then get it from claude plugin update conductor@conductor.

src/  mcp.ts         MCP server: one tool per command, every call through solve()  solve.ts       the loop: input check -> execute -> verify -> retry -> fallback  schema.ts      the JSON Schema subset used by command contracts  cc.ts          one hermetic `claude -p` call  verify.ts      verifier helpers: check/combine, zod schema, email/phone patterns, llmJudge  pipeline.ts    compose verified commands  router.ts      reliability-weighted command choice  telemetry.ts   JSONL log + per-command scores  library.ts     static command list (what the bundle ships)  registry.ts    directory scan used by the CLI and evals  cli.ts         manual driver  scores.ts      prints the scoreboardcommands/<name>/ index.ts (manifest + run? + verify) and tests/cases.jsoneval/            baseline-vs-conductor evals (paid)test/            node:test suite (free)dist/            committed bundle the plugin runs

Limitations and FAQ

• Model-backed tools need the claude CLI logged in, and each attempt is a separate claude -p process, so expect seconds per call and usage on your account. Pure tools need neither.
• Verifiers are mechanical. They catch format, grounding and budget violations, not every semantic error: categorize can pick an allowed but wrong category, and a summary can be within budget and still miss the point. No shipped command uses llmJudge yet.
• Some detectors are heuristics. redact_pii treats any long run of digits and separators as a phone number, so ISO dates or order numbers in the output fail the check and may get redacted on retry. summarize counts sentences by ., ! and ?, so abbreviations count.
• Telemetry stays on your machine (~/.conductor/telemetry.jsonl for the MCP server, projects/ for the CLI and evals). Inputs go nowhere except the claude -p calls themselves.
• Opening this repo as a Claude Code project shows a failed conductor server from .mcp.json; ${CLAUDE_PLUGIN_ROOT} only expands in an installed plugin. That is harmless.
• Not built yet: model-driven decomposition (pipelines are explicit), human gates for effectful commands, routing inside the MCP server (the client model picks tools itself), turning traces into new commands, and RL on the router.

License

MIT

More open source by Nir Feinstein

• agent-harness - make Claude Code or Codex act as a product team, a CEO or a portfolio manager
• frugal - your agent finds the cheaper honest way before it pays for anything
• flipback - an automatic /goal for every turn: sends Claude Code back when it stops early
• recall - your Claude Code memory in every agent, over MCP
• All projects