Luftpause

luftpause

The runtime layer under your agent harness. Every tool call is an await; the engine forgets the run, luftpause keeps it.

Tool calls split an agent into turns. To an inference engine each turn is a new request from nobody: when the tool returns, the continuation queues behind every batch request that arrived while it waited, and its KV cache may be gone. luftpause sits between the harness and the engines and keeps the run:

• Continuations go first. A run paused on a tool call resumes ahead of batch work; batch turns leave room for the interactive runs that will come back.
• A run stays where its prefix is. While a run is generating or paused on a tool, its turns go only to the replica that holds its KV cache; new turns are admitted by the engine's own queue.
• One caller cannot starve another. Held turns drain fairly across callers.
• Any harness, two doors. Point base_url at it (OpenAI-compatible, your own loop), or connect over MCP and get spawn / await sub-agents that survive a restart.

Measured on one 8 GB laptop GPU, engines saturated (48 batch runs flooding, 4 interactive runs beside them):

Try it

docker compose up -d                # fake engines, no GPU neededclaude mcp add --transport http --scope local luftpause http://localhost:7000/mcp

docker compose up -d pulls the published images; add --build to build them from source instead (first build takes a few minutes). It pulls latest; pin a release with LUFTPAUSE_VERSION=v0.1.4 docker compose up -d. The runtime is also listed in the official MCP Registry as io.github.vmedvedovskiy/luftpause.

Claude Code now has spawn, await, run, status, cancel, approve, tools and decide. The fake engines answer with filler, and the sub-agents' only tools are fake-mcp's lookup and deep_search, which wait a random second or so and return filler too — enough to see the fan-out. For real answers from an Ollama already on your machine, with Wikipedia as the sub-agents' tool instead of the fake ones:

docker compose -f compose.yaml -f compose.ollama.yaml up -d scheduler dataplane runtime

Wikipedia's API rate-limits anonymous callers, so a tool call can take tens of seconds while wikipedia-mcp retries. Llama 3.2 1B sometimes passes a tool's schema instead of its arguments; the runtime rejects the call and the sub-agent may answer without it, so ask again or pull a larger model.

Ollama needs the model pulled (ollama pull llama3.2:1b, or set OLLAMA_MODEL) and must listen beyond loopback (OLLAMA_HOST=0.0.0.0) unless Docker Desktop already reaches it (it does on Windows and macOS with the default loopback binding). Ollama serves one request at a time unless OLLAMA_NUM_PARALLEL is raised, so sub-agents queue behind each other there; luftpause sees no queue metric from it and does not cap it. SGLang on your GPU (compose.testbed.yaml) or any OpenAI-compatible endpoint (compose.runpod.yaml shows a hosted vLLM pod) work the same way.

How it differs

Routers place requests on a GPU pool; MCP servers delegate a sub-task to a local model. Neither knows what a run is. luftpause does: it sees every turn and every tool call of every run, through either door, and schedules them on one shared pool with three facts an engine never has: that a paused run will come back, what class of work it is, and who is asking.

The gains come from admission, not from routing. Routing alone ties SGLang's cache-aware router (below); with the engines idle, every arm ties. The difference appears when the engines are saturated, which is when it matters.

Two ways in

• OpenAI-compatible proxy — an agent that keeps its own loop points its base_url at luftpause. It sends x-luftpause-run-id per run and, optionally, x-luftpause-class and x-luftpause-caller; every tool call in its loop is then a scheduled await.
• MCP runtime — an agent such as Claude Code connects over MCP and gets spawn / await. luftpause runs each sub-agent's loop through the same proxy: LLM turns on the pool, tool calls to upstream MCP servers, the MCP session as the caller, and a journal so the runs outlive the process.

examples/ has three own-loop agents that set all three headers: the openai package, the OpenAI Agents SDK and LangGraph.

Connect from

A harness that cannot set a header per run still works through the proxy: the run is recognised by the tool_call_ids it echoes back, and a fixed x-luftpause-caller is enough for fairness.

Architecture

%%{init: {"themeVariables": {"fontFamily": "-apple-system, Segoe UI, Helvetica, Arial, sans-serif"}, "flowchart": {"wrappingWidth": 260}}}%%flowchart LR    mcp["Claude Code<br/>or any MCP client"]    own["Agents with<br/>their own loop"]
    subgraph lp["luftpause"]        runtime["<b>runtime</b> · .NET<br/>spawn · await · journal<br/>sub-agent loops"]        dp["<b>dataplane</b> · Rust<br/>OpenAI-compatible proxy"]        sched["<b>scheduler</b> · .NET<br/>decide(state, event)"]    end
    tools[("MCP tool<br/>servers")]
    subgraph pool["Engine pool"]        direction TB        engines["SGLang · vLLM"]        router["SGLang router"]        hosted["Hosted OpenAI-<br/>compatible APIs"]    end
    mcp -- MCP --> runtime    own -- "HTTP / SSE" --> dp    runtime -- "tool calls" --> tools    runtime -- "LLM turns" --> dp    dp -- "turns" --> pool    sched <-- "Admit · Report · run ended" --> dp    sched -. "health · load" .-> pool

• runtime (runtime/) — MCP server. Each spawned run is a .NET Task running the agent loop; tool calls go to the configured MCP servers. Every run is journaled to runs.jsonl and resumed on restart; a finished run is reported to the dataplane (DELETE /v1/runs/{id}) so the scheduler forgets it at once.
• dataplane (dataplane/crates/dataplane) — tokio/axum proxy. Resolves the run (header or tool-call ids), reads the class and caller headers, asks the scheduler where the turn goes, streams the engine's response through unchanged and reports first token, usage and finish reason.
• scheduler (scheduler/) — the decision core is pure functions over immutable state (Luftpause.Scheduling); the gRPC host executes the actions.
• test kit — fake-engine (OpenAI + /classify with simulated latency), fake-mcp (a lookup and a slow deep_search tool), tools/loadgen (agents, sub-agent fan-out over MCP, closed-loop throughput runs).

What it does

Failure semantics

What happens to a turn through the dataplane (/v1/chat/completions, /v1/decide) when something fails. Error bodies are {"error": {"message": "..."}}; every response except the scheduler-unreachable one carries x-luftpause-run-id. A failed turn frees its slot at once and leaves its run idle, never paused on a tool, so a retry is admitted like any next turn of that run. Retrying with the same x-luftpause-run-id keeps the run's affinity and fairness standing.

Constants

luftpause keeps few fixed numbers. There is no queue deadline and no turn limit: a held turn waits as it would in the engine's own queue, and a run ends when its tool budget is spent. What is left:

Running it

docker compose up -d --build                     # scheduler, dataplane, runtime, fake engines, fake MCP toolsdotnet run --project tools/loadgen -- --spawn 10 # fan 10 sub-agents out over MCP and await themdotnet run --project tools/loadgen -- --agents 8 --duration-s 60 --class batch --caller me # 8 own-loop agents through the proxy, closed looptools/check.sh                                   # both .NET test projects, loadgen, cargo test (when cargo is installed), image builds

Releasing: tools/release.sh vX.Y.Z builds, tags and pushes the three images to ghcr.io/luftpause (LUFTPAUSE_REGISTRY overrides) and creates the git tag; DRY_RUN=1 prints the commands only.

With a token: LUFTPAUSE_TOKEN=<token> docker compose up -d, then claude mcp add --transport http luftpause http://localhost:7000/mcp --header "Authorization: Bearer <token>".

An Ollama on the host as the only replica (model from OLLAMA_MODEL, default llama3.2:1b):

docker compose -f compose.yaml -f compose.ollama.yaml up -d scheduler dataplane runtime

On one GPU, two SGLang replicas of Llama-3.2-1B FP8 (the test bed used below):

docker compose -f compose.yaml -f compose.testbed.yaml up -d --build scheduler dataplane runtime fake-mcp sglang-a sglang-b

Traces in Jaeger: add compose.otel.yaml, which starts jaegertracing/all-in-one and points the three services' OTEL_EXPORTER_OTLP_ENDPOINT at it; a harness that sends traceparent on its MCP calls sees its sub-agents, their turns and the scheduler's admission inside its own trace. The UI is on http://localhost:16686.

docker compose -f compose.yaml -f compose.otel.yaml up -d --build

A hosted vLLM pod as an extra replica (key and pod id from the environment):

RUNPOD_POD_ID=<pod-id> RUNPOD_KEY=<key> docker compose -f compose.yaml -f compose.testbed.yaml -f compose.runpod.yaml up -d scheduler runtime

Measurements

One laptop, RTX 4060 Laptop 8 GB, two SGLang replicas of Llama-3.2-1B-Instruct FP8 unless stated. Repeats are noted per table; differences inside the spread are noise. The flood tables come from tools/bench/flood.sh (sub-agents over MCP) and tools/bench/proxy-flood.sh (own-loop agents); each takes the git revisions to compare as arms and builds them into their own image tags.

Fan-out: spawn × 16 + await all vs one sub-agent at a time (3 repeats)

16 sub-agents, ~2.8k-token tasks, lookup tool fixed at 2 s, up to 3 calls each.

Per sub-agent, fan-out and router take the same time (p50 15–22 s); the 1.4 s difference is inside the run-to-run spread.

Interactive runs under a batch flood (2 repeats, both shown)

48 batch runs flooding the replicas, 4 interactive runs beside them, 3 minutes per window, 8000-character contexts, up to 3 tool calls per run, closed loop (a finished run is replaced at once). Both engines queued requests in the no-scheduler arm, so the pool was saturated. SGLang runs with --enable-priority-scheduling in every row; no priorities are sent.

Agents with their own loop, base_url at luftpause (tools/loadgen --agents):

Sub-agents over MCP (tools/loadgen --spawn):

Interactive throughput is 2× the class-blind router through the proxy and 2.7× over MCP; batch gives up 9 % through the proxy and nothing over MCP. The rows differ in admission only. Letting runs move between replicas moved 39 of 48 batch runs mid-run and lost their prefix cache; pinning a generating or paused run to its replica is the whole gain. An estimate of each turn's KV memory on top of that, checked against the engine's reported pool, cost about a fifth of interactive throughput over MCP and nothing through the proxy, so it was removed. With light load (2000-character contexts, 24 batch runs) the engines never queue and all rows tie at about 14 interactive runs/min.

Claude Code as the client

Told to fan out six river look-ups, Claude Code spawned six runs and awaited them in one call: 12.5 s for the whole fan-out including its own round trips, against 36 s for the same runs back to back.

In a fresh session, asked only to "research these six rivers using the local sub-agents", it listed the sub-agent tools, restricted the runs to the Wikipedia tools, spawned six, awaited them in one call and re-spawned the weak answers. Tools: wikipedia-mcp over stdio.

The 70B pod sat in the same pool as the local 3B replica: turns asking for llama-3.3-70b went to the pod over https with its API key, everything else stayed local. failedToolCalls in await tells the caller when an answer did not come from tools.

Routing (N = 8 sub-agents in flight, 4 repeats)

Affinity equals the SGLang router's cache-aware routing and beats round-robin by 7–15 % at moderate load; at 16–24 runs in flight the caches churn and policy stops mattering.

Single-pass (vLLM classifier, RoBERTa-base)

luftpause serves two kinds of model on one pool. A generative engine (SGLang, vLLM, a hosted API) runs an agent's turns: it decodes token by token, keeps a KV cache per run, and a turn may end in tool calls that pause the run. A single-pass engine answers in one forward pass: an encoder or classifier (RoBERTa-style, served by vLLM's /classify) returns typed labels with probabilities, no decoding, no stream, no state between calls. Agents use it for the decisions that do not need a generated reply, such as routing a request, scoring a tool result or choosing the next step, at a few milliseconds instead of a generated turn. Sub-agents reach it through the decide MCP tool; own-loop agents through POST /v1/decide. Single-pass turns are placed on their own replicas and never enter run state, so they neither hold nor wait for a generative slot. For a label set the classifier was not trained on, decide(labels: [...]) scores the labels from a generative replica's first-token logprobs instead, at the cost of one one-token turn. On the 1B test bed it sorts "the server returned 500 three times and the on-call phone is ringing" into urgent at 0.89 and a population question into lookup at 0.99, but also calls "2 + 2" and "thanks, that's all" lookup: the mechanism works, the 1B model leans on the first label. Use a larger model or calibrate the scores.

License

Business Source License 1.1, converting to the Apache License 2.0 four years after each version is published. In practice: use it, change it, ship it inside your product or your company, redistribute it, all free, with no copyleft on your own code. The one thing you may not do until the conversion is offer luftpause itself to third parties as a hosted or managed service. That makes it source-available, not OSI open source, and we say so plainly. The reason is the usual one for infrastructure: a small team cannot compete with a cloud provider that wraps the project in a console and sells it, and that outcome would end the project rather than fund it. The change date is the answer to "will it ever be open source": yes, on a date written in the license, version by version. If the restriction blocks something you want to do before then, open an issue; a per-case grant is easy. Contributions are accepted under the same terms (see CONTRIBUTING.md).