Lm

Houtini LM (@houtini/lm) - Offload Work from Claude Code to a Local LLM, OpenAI GPT-5/6, a Router or a Cheaper Cloud Model

[npm version] [MCP Registry] [License: Apache 2.0] [Known Vulnerabilities] [Socket Badge]

Houtini LM is an MCP server that lets Claude (or any MCP client) hand bounded work to another model - a local LLM on your GPU, OpenAI's latest GPT models, a LiteLLM router, OpenRouter or a cheap cloud API - while you carry on working in the AI platform you already like. It cuts your token bill, and it gives you a second model to review your code whenever you want one.

[Houtini LM MCP server]

Quick Navigation

What's new | Why use it | Install | How it handles different models | What to hand over | Tools | Reading the footer | Configuration | Endpoints | The manual

I built this because I kept leaving Claude Code running overnight on big refactors and the token bill was painful. A huge chunk of that spend went on bounded tasks any decent model handles fine - generating boilerplate, code review, commit messages, format conversion, the sort of work that doesn't need Claude's reasoning or its tool access.

So Claude stays the architect, doing the planning, the multi-file changes and the judgement calls, and houtini-lm passes the drafting to whatever model you've got running. That could be Qwen on a GPU box under your desk, GPT-5 or GPT-6 straight from OpenAI, a model behind a LiteLLM router, one of OpenRouter's 300+ models or DeepSeek at pennies per million tokens. Claude QAs everything that comes back.

I wrote a full walkthrough of why I built this and how I use it day to day if you'd like the longer story.

What's new in 3.3

houtini-lm now speaks to far more than a local GPU. Point it at OpenAI directly and GPT-5, GPT-6 and the o-series reasoning models work without any router in between. They reject parameters every open model accepts (max_tokens, temperature), so houtini-lm sends them max_completion_tokens only, learns a model's output cap from its own error when the endpoint doesn't report it, and leaves the image, speech and moderation models out of the list. Point it at a LiteLLM router and it reads which real model sits behind each alias, along with that model's true context window and output cap, so a mixed fleet of local and hosted models is sized and profiled correctly. Thinking is now your call (auto, off or on), there's a Docker guide built from a working deployment, and the manual has a page per job. The changelog has the detail.

Why use houtini-lm?

The bottom line before we go deep: you keep your favourite AI platform, and you stop paying frontier prices for work that doesn't need a frontier model.

The obvious win is cost. When Claude delegates a review with code_task_files, the source files are read by the houtini-lm process and sent straight to the other model, so they never enter Claude's context window at all. Claude sends a short tool call and reads back a short answer. I benchmarked this on real TypeScript source files:

[Bar chart of Claude token use with and without delegation across four review tasks, averaging 93.3% saved]

That averages out at 93.3% saved across the session. To be fair, small tasks like a one-line question or a commit message don't save much, because the tool call overhead (around 250 tokens) is about the same size as the answer. Anything that involves reading files, which is most of a real coding session, pays for itself straight away. You can run the same benchmark against your own setup with LM_STUDIO_URL=http://your-server:1234 node scripts/benchmark.mjs.

The less obvious win is a second opinion. A different model reads your code with different blind spots, and it costs you next to nothing to ask. As it happens, the 3.3.0 release of this repo is a decent example: I had Claude point houtini-lm at gpt-6-astra (through my LiteLLM router) and ask it to review the release's own 14,000-token diff. It came back with three real bugs - a router probe that cached a temporary 429 as a permanent "not a router", an output cap that got overwritten, and a race in the model list cache - and all three were fixed before the release shipped. Two of the three new test files were drafted the same way, then reviewed by Claude before commit.

Code review is where this pays off hardest, because reviews are exactly the bounded, read-a-lot-write-a-little work that a cheaper model does well. From there the list keeps growing: test stubs, docstrings, commit messages, changelog drafts, format conversion, mock data, type definitions, embeddings for a RAG pipeline, a quick sanity check on a regex, brainstorming three approaches before Claude commits to one, and so on.

The trade-off is wall-clock time. Local inference is typically 3-30x slower than a frontier model, so delegation wins on bounded, self-contained tasks rather than everything. A local model keeps your code private and costs nothing per token, a cloud model is cheap, and neither touches your Claude quota or its rate limits.

How it works

Claude Code, Claude Desktop, Cursor... (orchestrator)   |   |-- Reasoning, planning, architecture, tool use --> your main AI platform   |   +-- Bounded grunt work --> houtini-lm --HTTP/SSE--> any OpenAI-compatible endpoint       . Code review & second opinions       LM Studio, Ollama, vLLM, SGLang, llama.cpp       . Test stubs & boilerplate            LiteLLM routers       . Commit messages & docs              OpenRouter (300+ models)       . Format conversion & mock data       DeepSeek, Groq, Cerebras, OpenAI...       . Embeddings for RAG pipelines

Claude's the architect, the other model's the drafter, and Claude checks everything that comes back.

Install

You'll need Node 22.5 or newer and an OpenAI-compatible endpoint: LM Studio, Ollama, vLLM, SGLang, a LiteLLM router, or a cloud API key for OpenAI, DeepSeek, Groq and the like. In Claude Code, with LM Studio running on the same machine, it's one command:

claude mcp add houtini-lm -- npx -y @houtini/lm

That's it. LM Studio listens on localhost:1234 by default, which is where houtini-lm looks first, so Claude can start delegating straight away. Anywhere else, set the URL (and a key, if the endpoint needs one):

claude mcp add houtini-lm \  -e HOUTINI_LM_ENDPOINT_URL=http://192.168.1.50:1234 \  -e HOUTINI_LM_API_KEY=your-key-if-needed \  -- npx -y @houtini/lm

OpenAI works the same way. Pin the model you want, because OpenAI lists dozens and they all score the same in routing:

claude mcp add houtini-lm \  -e HOUTINI_LM_ENDPOINT_URL=https://api.openai.com \  -e HOUTINI_LM_API_KEY=sk-... \  -e HOUTINI_LM_MODEL=gpt-5.2 \  -e HOUTINI_LM_SERIALISE=0 \  -- npx -y @houtini/lm

Installing houtini-lm walks through every route: a GPU on another machine, OpenAI and other cloud APIs, OpenRouter, a LiteLLM router, Claude Desktop and other MCP clients, plus how to check it worked and how to update. If you'd rather run it in a container, Running houtini-lm in Docker covers both a plain docker run -i and serving it over HTTP behind Docker's MCP Gateway. New to local models altogether? Start with Getting started, which covers which models fit on 16, 32, 64, 96 or 128 GB of VRAM.

To check everything's wired up, ask Claude to run houtini-lm's discover tool. It tells you the version, which endpoint it found, which model is active and how big its context window is.

How houtini-lm handles different models

No two open source LLMs are the same. They differ in context window, output cap, prompt template, whether they think before they answer and how much of that they report, so a lot of houtini-lm's code is about working out what it's talking to and adjusting for it. How houtini-lm handles different models has the full detail, and here's the short version.

At startup houtini-lm asks your server for every model it has, loaded and downloaded, and looks each one up on HuggingFace for its architecture, licence and chat template, caching the lot in SQLite so later startups are instant. For the families I know well (Qwen, Nemotron, Granite, LLaMA, GLM, GPT-OSS, DeepSeek, Gemma, Kimi and OpenAI's hosted GPT models) there's a curated profile, and each family gets its own temperature, output constraints and think-block handling, while OpenAI's hosted reasoning models (GPT-5/6, o-series) are sent only the parameters they accept. Run list_models and you get the whole picture:

Loaded models (ready to use):
  nvidia/nemotron-3-nano    type: llm, arch: nemotron_h_moe, quant: Q4_K_M, format: gguf    context: 200,082 (max 1,048,576), by: nvidia    Capabilities: tool_use    NVIDIA Nemotron: compact reasoning model optimised for step-by-step logic    Best for: analysis tasks, code bug-finding, math/science questions    HuggingFace: text-generation, 1.7M downloads, MIT licence
Available models (downloaded, not loaded):
  qwen3-coder-30b-a3b-instruct    type: llm, arch: qwen3moe, quant: BF16, context: 262,144    Qwen3 Coder: code-specialised model with agentic capabilities    Best for: code generation, code review, test stubs, refactoring    HuggingFace: text-generation, 12.9K downloads, Apache-2.0

Output budgets come from the model each call is actually sent to. Leave max_tokens unset and the call gets 25% of that model's context window, never more than its declared output cap or the room left beside your prompt, so a hosted model doesn't get sent a request it'll reject. There's a floor too, because MCP clients habitually pass tiny caps like 256 that strangle reasoning models mid-thought, so anything under 4,096 is ignored unless you set HOUTINI_LM_MIN_TOKENS=0.

Thinking is your decision, through HOUTINI_LM_THINKING. The default, auto, switches thinking off for models detected as supporting the toggle (Qwen3, Gemma 4, Nemotron, DeepSeek R1, GLM-4, gpt-oss), which suits Claude doing the reasoning and the other model doing the drafting. off forces that on every call, which you need when a backend serves a thinking model under a name detection can't recognise. on forces thinking on, which is worth it for bug-hunting or checking an argument, at the cost of time and tokens. Whichever you choose, houtini-lm inflates the output budget for thinking models and strips any <think> blocks from the answer, so the reasoning doesn't leave you with an empty reply.

With several models available, houtini-lm scores each against the task type and picks the best, and it suggests a better model rather than swapping one in, since loading a model takes minutes. On a big catalogue every unknown model scores the same and the first listed wins, so pin one with HOUTINI_LM_MODEL, or pass model on an individual call.

Point it at a LiteLLM router and it reads /model/info as well, which tells it the real model behind each alias (my local alias is qwen3.6-27b) and, for hosted models, the true context window and output cap. Each alias is then profiled and sized as the model it actually is, the TTS, image and video models a router lists by the dozen are filtered out, and rate-limit errors are retried with backoff. Calls are queued one at a time by default, because a single GPU can only serve one request anyway; OpenRouter skips the queue, and for OpenAI, a router or a batching backend like vLLM you can turn it off with HOUTINI_LM_SERIALISE=0.

What to hand over

The best candidates are bounded and well defined, with a clear input and a clear output:

Anything that needs reasoning across the codebase, tool access or multi-step orchestration stays on Claude: architectural decisions, reading and writing files, running tests and interpreting the results, multi-file refactoring plans and anything that has to call other tools. The tool descriptions are written to nudge Claude into planning delegation at the start of a big task, rather than only using it when it happens to remember.

The tools

There are eight of them. The full parameter reference is in The tools, in depth.

The inference tools (chat, custom_prompt, code_task, code_task_files) all take an optional model to pin the call, max_tokens, and sampling controls (temperature, seed, stop, top_p, top_k, repeat_penalty, frequency_penalty, presence_penalty). chat and custom_prompt also take a json_schema, which forces the answer to conform to a JSON Schema; on LM Studio that's grammar-based sampling, so there's no hoping the model remembers to close its brackets:

{  "json_schema": {    "name": "code_review",    "schema": {      "type": "object",      "properties": {        "issues": {          "type": "array",          "items": {            "type": "object",            "properties": {              "line": { "type": "number" },              "severity": { "type": "string" },              "description": { "type": "string" }            },            "required": ["line", "severity", "description"]          }        }      },      "required": ["issues"]    }  }}

If you're driving houtini-lm from your own scripts rather than from Claude, set HOUTINI_LM_STRUCTURED=1 and each inference result also carries a structuredContent block (the answer, model, tokens, timing, quality flags) to read as JSON. Leave it off for Claude Code, which shows the model only that block when it's present (the tools page has the story).

Reading the footer

Every response ends with a footer computed from the SSE stream itself:

---Model: nvidia/nemotron-3-nano | 279→303 tokens (12 reasoning / 291 visible) | TTFT: 485ms, 58.0 tok/s, 5.2s📊 First measured call on nvidia/nemotron-3-nano: 58.0 tok/s, 485ms to first token - use this to gauge whether to delegate longer tasks.💰 Offloaded - this session: 4,283 tokens / 7 calls · lifetime: 147,432 tokens / 213 calls

The first-call line appears once per model per session, and it's a benchmark from a real task rather than a synthetic warm-up. The 💰 line updates every call, and it counts the tokens the other model handled (its prompt and completion, reasoning included), which is work Claude didn't do rather than a measure of Claude tokens saved; the benchmark above is the honest measure of that. When a model reports its reasoning tokens, the token count splits into reasoning and visible, so you can see when a thinking model is burning budget on hidden reasoning.

When something went wrong, a quality line says so: TRUNCATED for a partial result (a stalled connection gives you what arrived rather than a timeout error), hit-max-tokens when the budget ran out, think-blocks-stripped when reasoning was removed and tokens-estimated when the server didn't report usage. Clean output gets no quality line at all.

Per-model speed and token counts persist in ~/.houtini-lm/model-cache.db, so discover shows your measured tok/s and time to first token from the first call of a new session, and stats gives you the lifetime totals. That data is specific to your workstation on purpose, because delegation decisions should reflect your hardware rather than somebody else's benchmark. In practice, Claude delegates more the longer a session runs; after about 5,000 offloaded tokens it starts hunting for more work to push over.

Getting good results

Qwen, Llama, Nemotron and GLM score brilliantly on coding benchmarks now, and the gap between a good and a bad result is almost always the prompt rather than the model. I've spent a fair bit of time on this, and the short version goes like this. Send complete code, because local models make up details when the input is truncated, so send the whole function rather than a snippet with ... in the middle. Be explicit about the output format ("return a JSON array"), since smaller models need it. Give the model a specific persona ("expert Rust developer who cares about memory safety" does noticeably better than "helpful assistant"), state what not to do as well as what to do, and for code generation include the imports, types and signatures around the function body.

Keep long jobs in chunks, too. Most MCP clients time a tool call out at around 60 seconds, and although houtini-lm sends a progress notification on every streamed chunk to keep the clock reset, not every client or gateway passes those through. Calls of roughly 500-900 output tokens finish comfortably inside the limit. The craft of delegation goes much deeper, including the verbatim-echo pattern I use for fixes.

Check your install

npm run shakedown

scripts/shakedown.mjs runs seven of the eight tools end to end (everything except stats) and prints a table of real TTFT, tok/s, token counts and reasoning split for each call. It takes under a minute on a decent rig:

Summary
   7/7 steps passed on LM Studio, model=nvidia/nemotron-3-nano
| Tool              | OK  | TTFT (ms) | tok/s  | Tokens in→out        | Reasoning | Notes| chat              | ✅  |      891  |   36.9 | 48→104               |        —  | answered| custom_prompt     | ✅  |      872  |   43.9 | 170→333              |        —  | 5 valid items| code_task         | ✅  |      857  |   41.6 | 180→189              |        —  | tests generated| code_task_files   | ✅  |   11028   |   39.5 | 6891→3000            |        —  | cross-referenced| embed             | ✅  |      —    |     —  | —                    |        —  | 768-dim vector
   Tokens offloaded: 10,915 (prompt: 7,289, completion: 3,626, reasoning: 0)

If you'd rather have a quality review than latency numbers, paste SHAKEDOWN.md into a Claude session with houtini-lm attached and Claude will drive the same steps and write you a report on the output as well as the speed.

Configuration

Most setups need only the first two or three of these. The full list, including the file-access and queuing controls, is in Configuration.

Compatible endpoints

Anything that speaks the OpenAI /v1/chat/completions API will work:

The manual

This README is the overview, and the depth lives in these pages:

Development

git clone https://github.com/houtini-ai/houtini-lm.gitcd houtini-lmnpm installnpm test             # build + unit testsnpm run shakedown    # end-to-end self-test against a live endpoint

DEVELOPER.md covers the architecture, the reasoning-model pipeline, backend detection, the SQLite performance cache and how to add new tools or backends. If you find a model family houtini-lm handles badly, open an issue with the discover output and I'll take a look.

Good luck, and let me know how you get on!

Licence

Apache-2.0