MCP Failure Lab

MCP Failure Lab

[OpenSSF Scorecard] [OpenSSF Best Practices]

[DOI]

A deliberately faulty MCP server and a bounded reference-client harness. v0.1.0, MIT. Built on the official MCP TypeScript SDK 2.3.1, not a replacement protocol implementation.

Try it

Requires Docker. The image uses Node 24 and pinned npm dependencies.

docker build -t mcp-failure-lab:local .docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges mcp-failure-lab:local npm testdocker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges mcp-failure-lab:local node runner.mjs

The last command prints JSON results and OpenAI-message-shaped traces from seven deterministic runs. No language model participates. Timings are observed elapsed milliseconds, not benchmarks. The test command exits nonzero on a regression; the demo command emits observations, not a grading verdict.

For local development: npm ci --ignore-scripts, then npm test.

From the published image (Official MCP Registry: io.github.cloudlabworks/mcp-failure-lab):

docker run -i --rm -e LAB_SCENARIO=tool-error ghcr.io/cloudlabworks/mcp-failure-lab:0.1.2

Verifying a release

From v0.1.2, each release has build provenance signed through Sigstore by the publish-mcp workflow. With the GitHub CLI:

gh attestation verify oci://ghcr.io/cloudlabworks/mcp-failure-lab:0.1.2 --repo cloudlabworks/mcp-failure-labgh attestation verify mcp-failure-lab-0.1.2.tar.gz --repo cloudlabworks/mcp-failure-lab

The second command checks the source tarball from the GitHub Release. To verify offline, add --bundle mcp-failure-lab-0.1.2.tar.gz.intoto.jsonl (the provenance file attached to the same release).

Faults

Only the advertised lab_probe tool is affected. Initialization and discovery use the official SDK. The transport wrapper corrupts result shape, not JSON framing; truncated frames, bad IDs, HTTP failures and authentication faults are not implemented.

Configuration is read at process startup. Scenarios reset per process. LAB_DELAY_MS: integer 0–10000 (default 1000). LAB_FAILURES: integer 0–10 (default 2). Values must be plain decimal digits; an unset or empty variable uses the default, and anything else (0x10, 1e3, 7) stops the server with exit code 2. The reference client retries only explicit tool errors, at most five attempts; protocol errors, timeouts and disconnects stop immediately. The harness's tool-call deadline is 10–10000 ms; SDK connection/discovery deadlines are separate. Retries intentionally have no backoff so the generated fixture exposes repeated calls; this is not recommended production retry policy.

Attach your own MCP client

Configure a test MCP connection to run this command, with stdin/stdout attached:

docker run --rm -i --network none --read-only --cap-drop ALL --security-opt no-new-privileges -e LAB_SCENARIO=transient-error -e LAB_FAILURES=2 mcp-failure-lab:local

Call lab_probe with {"message":"probe"}. There are no real downstream services, credentials, file-writing tools or listening network ports. Client integrations beyond the included reference client are unverified.

Feed a failure into Agent Trace Clinic

From this repository, with Clinic checked out next to it:

mkdir -p artifactsdocker run --rm --network none --read-only mcp-failure-lab:local node runner.mjs > artifacts/run.jsonnode --input-type=module -e 'import fs from "node:fs"; const r=JSON.parse(fs.readFileSync("artifacts/run.json","utf8")); console.log(JSON.stringify(r.results.find(x=>x.scenario==="tool-error").trace));' > artifacts/tool-error.jsonnode ../agent-trace-clinic/cli.mjs artifacts/tool-error.json --html > artifacts/clinic-report.html

Clinic should identify tool_loop, retry_storm and unhandled_tool_error on this fixture. These are heuristic leads: deliberate bounded retries can also trigger them. No token counts are invented.

Relationship to upstream

MCP conformance is the upstream authority for protocol tests. This project contributes a small fault surface for recovery experiments; it does not claim conformance certification. The official framework currently documents HTTP-based client/server scenario runners. This first release is stdio-only; an upstream scenario adapter and a Streamable HTTP target are next, not shipped capabilities.

Sources checked 2026-10-14:

• SDK v2 documentation
• SDK source and examples
• Conformance framework

Maintenance contract

One transport, one tool, six fault modes, one reference client. Pin SDK upgrades and rerun every fault test. Owner: Cloud Lab Works / Waku. Gherkin review has not been assigned or completed. Before public release: source/dependency review, an independent test pass, and publication approval. Before cloud execution: bounded job budget, teardown, and measured runtime. No cloud account is required for this release. No telemetry or remote reporting. Generated reports stay in ignored artifacts/.

Agent Reliability Lab

Open the public demo or read the experiment guide. Four controlled before/after cases connect the official MCP reference client to deterministic Clinic inspection. The public sample is recorded local evidence, not a live cloud execution. Cloud adapters and runbook are included.

Citation

If you use this project, please cite the accompanying preprint: MCP Failure Lab: Reproducible Tool-Failure Fixtures and Cross-Platform Traces for Testing Agent Reliability. DOI: 10.5281/zenodo.23720942. See the paper page for the abstract and PDF.

@techreport{alvord2026mcp,  author = {Alvord, Alex},  title = {{MCP Failure Lab: Reproducible Tool-Failure Fixtures and Cross-Platform Traces for Testing Agent Reliability}},  institution = {CloudLab Works},  publisher = {Zenodo},  year = {2026},  month = oct,  type = {Preprint},  doi = {10.5281/zenodo.23720942},  url = {https://doi.org/10.5281/zenodo.23720942}}