API Shakedown

API Shakedown

Give it an OpenAPI spec. Get back a ready-to-run test harness: WireMock stubs, contract tests, robustness probes, and chaos scenarios, written by a team of LLM agents that review each other's work.

[CI] [PyPI] [Python] [License: MIT]

Runs locally on Ollama by default: no API key, no data leaves your machine. Cloud models (OpenAI, Anthropic) are bring-your-own-key. Available as a CLI, a Docker image, and an MCP server so an AI assistant can drive it.

Status: alpha (0.1.3). Works end to end on the example spec with a local 8B model.

[Demo: shakedown run generates a harness, its contract tests pass, and a chaos drill makes the first two calls fail with 503 before recovering]

A real run on the bundled example with qwen3:8b on a laptop, sped up: generation took about 9 minutes. Recording: docs/demo.cast.

Contents

• Why
• Where it helps
• Quick start
• What you get
• How it works
• Install
• Usage
• Configuration
• MCP server
• Docker
• Architecture
• Limitations
• Roadmap
• Development
• Contributing
• License

Why

Teams that consume an API need three things that are tedious to write by hand and go stale quickly:

1. A realistic mock of the API to develop and test against.
2. Contract tests that tell you when the real API stops matching its spec.
3. Failure drills: what does your client do when the API is slow, flaky, rate limited, returns garbage, or goes down?

API Shakedown generates all three from the spec you already have, and keeps them honest: the LLMs never write code that runs. They propose data (response bodies, test cases, failure scenarios, test records), and code validates every item against the spec, feeds problems back to the model, and falls back to deterministic output when a model cannot get it right. Without any model at all (--offline), every part still works.

Where it helps

It is not a load-testing tool or a security scanner, and the virtual service does not keep state between calls; see Limitations.

Quick start

Prerequisites: Python 3.11+, Docker (for WireMock), and Ollama with a model (ollama pull qwen3:8b) unless you use --offline or a cloud model.

pip install api-shakedownshakedown doctor                                   # is the model ready?shakedown run openapi.yaml --out my-harness        # minutes on a local modelcd my-harnessdocker compose up -d                               # WireMock with your stubs on :8080pip install -r requirements.txtSHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.pypython chaos/chaos.py on flaky-start               # now point your client at :8080python chaos/chaos.py off

Try it on the bundled example: shakedown run examples/petstore.yaml --out petstore.

A real run on the example spec with qwen3:8b:

Petstore 1.0.0: 5 operations (ollama / qwen3:8b)Generating test data...Test data ready (3 Pet).Generating stubs, contract tests, and chaos scenarios...  stubs: done  contracts: done  chaos: done  review contracts: approved  review stubs: approved  review chaos: approved
Wrote 14 files to harness in 436s: 15 stubs, 11 contract cases, 11 chaos scenarios, 30 probes.Next: cd harness && docker compose up -d   (then see README.md)

What you get

my-harness/├── README.md                  # generated: how to run it, every scenario, review notes├── docker-compose.yml         # WireMock 3 serving the stubs (port: SHAKEDOWN_WIREMOCK_PORT)├── wiremock/mappings/         # stub mappings├── tests/│   ├── test_contract.py       # contract tests (cases in contract_cases.json)│   └── test_robustness.py     # malformed-request probes (cases in robustness_cases.json)├── chaos/│   ├── chaos.py               # switch failure scenarios on/off (standard library only)│   └── chaos_scenarios.json├── data/dataset.json          # test records and edge-case values per field├── requirements.txt           # pytest, httpx, openapi-schema-validator└── shakedown.json             # manifest: counts, review outcome, generation notes

Baseline chaos scenarios:

LLM-targeted scenarios from a real run include createPet fails 3 times with 503 then recovers, listPets is rate limited, and getPet loses required fields.

How it works

flowchart LR    spec[/"OpenAPI spec"/] --> load["Load, validate,<br/>normalize, chunk"]    load --> data["Test data agent"]    data --> stubs["Stub agent"]    data --> contracts["Contract agent"]    data --> chaos["Chaos agent"]    stubs --> review{"Reviewer agent"}    contracts --> review    chaos --> review    review -- "blocker: revise with feedback" --> stubs    review -- "blocker" --> contracts    review -- "blocker" --> chaos    review -- "approved, or budget spent" --> coord["Coordinator"]    coord --> harness[/"Harness directory"/]

1. Spec layer: loads YAML/JSON from a path or URL, validates it (problems become warnings), inlines $refs safely, and splits operations into prompt-sized chunks so large specs fit small models.
2. Test data agent runs first: the stubs and tests share its records.
3. Stub, contract, and chaos agents run in parallel.
4. Reviewer judges each artifact. Blockers send it back to its author with the feedback, for a bounded number of rounds; suggestions are kept as notes.
5. Coordinator assembles the harness and its README, including anything the reviewer still rejected when the revision budget ran out.

Every agent follows the same pattern, which is what keeps small local models reliable:

sequenceDiagram    participant A as Agent    participant M as LLM    participant C as Code checks    A->>M: task + JSON schema (constrained decoding on Ollama)    M-->>A: proposal (bodies, cases, scenarios, records)    A->>C: validate against the spec    alt problems found        C-->>A: exact errors, e.g. "listPets 200: 'id' is a required property"        A->>M: fix these problems        M-->>A: corrected proposal    end    A->>A: keep valid items, fall back to deterministic ones for the rest

Things code guarantees, so the model never has to:

• Request matching, URL patterns, priorities, and path-parameter types in the stubs.
• Response bodies valid against their schemas.
• Test inputs that route (valid path parameters, required query parameters), success cases that use ids that exist, 404 cases that use ids that do not.
• Authentication from the spec's security schemes.
• Chaos scenarios that are internally consistent (a "flaky" failure must recover, a rate limit is a 429, a partial outage leaves some endpoints up).
• Reviewer findings that cite real items, and blockers only on what the author controls.

Install

pip install api-shakedown              # local models via Ollama; includes the MCP serverpip install "api-shakedown[cloud]"     # plus OpenAI and Anthropic

Or use the Docker image, which includes everything: see Docker.

Generated harnesses need Docker (for WireMock) and Python for their tests.

Usage

shakedown doctor                     Check the LLM provider is reachable and readyshakedown inspect SPEC [--json]      Show the operations the agents will seeshakedown run SPEC [options]         Generate a harnessshakedown version

shakedown run options:

Using the harness:

docker compose up -d                                     # stubs on http://localhost:8080SHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.py  # against the stubspython chaos/chaos.py list | on <scenario> | off         # failure drillsSHAKEDOWN_TARGET_URL=https://api.example.com/v1 pytest tests   # against a real serverSHAKEDOWN_SEED=1 SHAKEDOWN_TARGET_URL=... pytest tests   # create the test records first

With SHAKEDOWN_SEED=1, the contract tests first create the test dataset's records on the server, using the spec's own create operations, and delete them when they finish. This writes to the server, so point it at a test instance.

PowerShell: $env:SHAKEDOWN_SELECT_STATUS = "1"; pytest tests/test_contract.py.

Configuration

Settings come from environment variables or a .env file (see .env.example). Everything is optional.

Mixing models works well: generate locally, review with a stronger cloud model:

SHAKEDOWN_AGENT_MODELS='{"reviewer": "anthropic:claude-sonnet-5-5"}' shakedown run openapi.yaml

Generated harnesses read SHAKEDOWN_TARGET_URL, SHAKEDOWN_SELECT_STATUS, SHAKEDOWN_SEED, SHAKEDOWN_API_KEY, SHAKEDOWN_BASIC_AUTH, SHAKEDOWN_TIMEOUT, and SHAKEDOWN_WIREMOCK_PORT / SHAKEDOWN_WIREMOCK_URL.

MCP server

shakedown-mcp (stdio) lets an MCP client such as Claude Code or Claude Desktop generate harnesses and run failure drills:

claude mcp add api-shakedown -- uvx api-shakedown mcp     # Claude Code, nothing to installclaude mcp add api-shakedown -- shakedown-mcp             # Claude Code, after pip install

Name in the official MCP Registry: io.github.anzar-ahsan-commits/api-shakedown.

Other clients (mcpServers in their config):

{  "mcpServers": {    "api-shakedown": {      "command": "uvx",      "args": ["api-shakedown", "mcp"],      "env": { "SHAKEDOWN_PROVIDER": "ollama" }    }  }}

Pass absolute paths. Generating with a local model takes minutes; if your client gives up on long tool calls, use offline or review: false, or raise its tool timeout.

Docker

docker build -t api-shakedown .docker run --rm -v "$PWD:/work" api-shakedown run /work/openapi.yaml --out /work/harnessdocker run -i --rm -v "$PWD:/work" --entrypoint shakedown-mcp api-shakedown   # MCP server

The image reaches Ollama on the host at host.docker.internal. Details for Linux, PowerShell, and cloud keys: DOCKER.md.

Architecture

flowchart TB    subgraph app["api-shakedown"]        cli["CLI: shakedown"] --> pipeline["pipeline"]        mcp["MCP server: shakedown-mcp"] --> pipeline        pipeline --> agents["agents: testdata, stubs, contracts,<br/>chaos, reviewer, coordinator"]        agents --> domain["spec, wiremock, contracts,<br/>chaos, testdata (code checks)"]        pipeline --> harness["harness writer + templates"]    end    subgraph core["agentpanel (reusable core)"]        workflow["Workflow: parallel specialists,<br/>cross-review, bounded revision"]        structured["structured output:<br/>validate, feed back, retry"]        llm["providers: Ollama, OpenAI, Anthropic"]    end    agents --> workflow    agents --> structured    structured --> llm

The orchestration core is published separately as agentpanel: a small, domain-agnostic library for "specialists draft, reviewers critique, authors revise, a coordinator decides" workflows on LangGraph. It never imports the application; a test enforces that.

packages/agentpanel/     # the reusable core (its own PyPI package)src/api_shakedown/       # the application  spec/                  # loading, validation, $ref resolution, edge cases, samples  agents/                # the six agents  templates/             # runners copied into every harnessexamples/petstore.yaml   # example spectests/                   # offline tests; `-m live` for Ollama + Docker tests

Limitations

• Speed on local models: a full run on the 5-operation example takes about 7 minutes with qwen3:8b, most of it review; --no-review takes about 2.5 minutes.
• Small reviewers are imperfect: an 8B reviewer sometimes raises a false blocker, which costs a revision round. Use a stronger model for the reviewer if you can.
• Specs: OpenAPI 3.0-3.2. Swagger 2.0 must be converted first; references to other files are not followed; self-referencing schemas are not validated.
• Stubs are static: a create call returns a new record, but stubs do not keep state between calls.
• Probes against stubs only cover operations that document a 4xx; they are meant for real servers.
• Seeding has limits: SHAKEDOWN_SEED=1 can only create records the spec lets it create with the test data's ids, and real servers often enforce rules the spec doesn't state. Records it can't create are listed in a warning, and tests that read them fail with 404 until you create them yourself.
• Scope: not a load-testing tool or a security scanner.
• Large specs are slow on a laptop: the Typesense API (79 operations) takes several hours end to end with qwen3:8b on a laptop GPU, versus minutes for small specs. A bigger GPU or a cloud model is much faster; --offline takes seconds. Reports from more real-world specs are very welcome.

Roadmap

• Faster runs on large specs (smaller prompts per batch, progress per batch)
• Smarter seeding: capture server-assigned ids from create responses, so resources like API keys can be seeded too
• Stateful stubs (WireMock response templating, created records visible to later calls)
• Auth-failure scenarios (401/403) and "valid but nasty input must not 5xx" probes
• Review of the test dataset
• More harness targets (k6 load profiles, Postman collections)

Development

git clone https://github.com/anzar-ahsan-commits/api-shakedown.gitcd api-shakedownpython -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activatepip install -e packages/agentpanel -e ".[dev]"ruff check . && ruff format --check . && mypypytest                 # offline: LLMs are faked, about 20 spytest -m live         # also uses your local Ollama and Docker

Contributing

Contributions are welcome: see CONTRIBUTING.md. Security issues: see SECURITY.md.

License

MIT © 2026 Anzar Ahsan