Model Manager

mcp-modelmanager

Manage your own local model machine through a language-model tool (Claude, ChatGPT, etc.). The models and their data stay on your hardware; the server is designed so the machine need not be opened to the internet by this software.

This MCP server gives an AI session controlled access to your own Ollama/model machine: read state, pull and remove models, switch the inference service to a different model, build custom Ollama variants and run LoRA training. The models run on your hardware. The server process itself talks only to your own computer or local network, never to a public address, on its own. What it does knowingly reach out to a public address for is a model: calling fetch_model makes the model machine connect to Hugging Face at runtime to download it. That is a deliberate, tool-triggered outbound connection, not something the server does behind your back or only once at setup. switch_service never reaches Hugging Face itself; if the model is not yet on the machine, it refuses and tells you to run fetch_model first.

What that does and does not promise. The inference itself stays local: your prompts to the managed models and their answers do not go to any cloud model provider. But the language model you drive this server with (Claude, ChatGPT, etc.) is itself a hosted service in the usual case. Your instructions to it and the tool results it sees pass through that provider like any other MCP call. "Local" here means the managed machine and its data; it is not a claim that nothing you type ever reaches a third party. Choose the driving model accordingly, or drive it with a local one.

Operating systems: The plain query path (reading status and models over the tunnel) runs cross-platform. Management and setup (service and training control) require a POSIX system with systemd (systemd user unit, ssh -F /dev/null); Windows is not supported for management.

The server ships with no address of its own and no default values for a machine. You enter where your machine is; without those details the server deliberately does not start (see Required fields and SETUP.md).

What the server can and cannot do

It can: query state (GPU memory, disk space, running service, containers), list/pull/remove models, switch the vLLM inference service to a different model (with an up-front estimate of the memory need), build custom Ollama variants (fixed system prompt, parameters, later an adapter) and start/track/cancel LoRA training.

It cannot and deliberately will not: run an arbitrary command on the machine. There is no tool for free remote control. Every operation is a fixed template; every parameter first passes an allowlist (see Security model).

It is not a query bridge. For merely asking a model you use a separate, read-only MCP server (e.g. a simple HTTP bridge to Ollama). This server manages the machine. Two separate servers ensure that a query call never accidentally holds management rights.

Your machine: three possible cases

The model machine can be attached in different ways. All three cases are supported; you only enter a different address.

Safeguard: the server process's own MCP/SSH/HTTP traffic goes only to the own computer or the local network. Addresses outside (public IPs, foreign hostnames) are rejected as MM_VM_HOST, so the server does not reach out to a public address on its own to talk to your model machine. For case c) this means: the tunnel/VPN ends locally or in a private address range, and you enter that local/private address, not the public address of the far side. This safeguard does not cover model downloads: fetch_model makes the model machine itself reach out to Hugging Face, a public address, whenever you ask it to fetch a model (see the caveat above); switch_service refuses a model that is not already on the machine instead of fetching it. fetch_model(source="ollama") sends the machine to a second, equally deliberate public address: it makes the machine's Ollama daemon pull the model via /api/pull, which reaches out to registry.ollama.ai for plain model names; a prefixed name such as hf.co/user/repo sends it to that other registry instead.

The plain query (state, version, loaded models) runs over HTTP to the service port. Management (pulling models, switching the service, training) runs over a secured SSH control channel. For management you therefore need SSH access to the machine, also in case a) (then SSH to 127.0.0.1).

Installation

Requirement: Python 3.11 or newer.

With uv (recommended):

uv venv --seed .venv.venv/bin/pip install -r requirements.txt

--seed ensures pip is present in the venv.

With Python built-ins (fallback):

python3 -m venv .venv.venv/bin/pip install -r requirements.txt

Should python3 -m venv produce a venv without pip on your system (missing ensurepip), use the uv path above.

There are only two direct dependencies, both pinned: the MCP SDK (mcp==1.30.0) and an HTTP client (httpx==0.28.1). These two pins live in requirements.txt and pyproject.toml; the full resolved dependency tree (including the transitive packages anyio, pydantic, starlette, uvicorn, jsonschema, pyjwt) is additionally pinned in uv.lock. CI checks with uv lock --check that the lock still matches pyproject.toml, then installs the exact locked tree for the tests, so a drifted lock fails CI. A separate, advisory pip-audit job checks the locked tree for known vulnerabilities on every push to main and pull request and again weekly; it does not block a pull request. The SSH access uses the system's ssh program.

Required fields

The server reads its configuration exclusively from environment variables. These three you must set, otherwise it does not start:

If one is missing, the start aborts with a clear message that says which field is missing. Example:

$ PYTHONPATH=src .venv/bin/python -m mcp_modelmanager.serverMM_CONTAINER_ROOT is not set. Enter the root directory on your model machineunder which all model, adapter and cache data live. Without this value theserver does not start. See SETUP.md.

All further fields have sensible defaults (Ollama and vLLM standard ports, key storage in your own ~/.ssh) and are only set when needed; see Environment variables.

The full step-by-step setup from the empty state to the running server (including the secured SSH channels) is in SETUP.md. Building the GPU model machine itself (an Ollama and/or vLLM install reachable under the MM_* variables below) is the machine-side task, and it ships with this repo: the operator-side scripts (setup/, vm_side/), the network fence templates (security/) and the machine-side guides (MACHINE.md, SECURITY.md, SETUP-MACHINE.md) sit next to the server here. Start from MACHINE.md for the machine side.

MCP registration (example with placeholders)

Register the server with your language-model tool. Replace the placeholders in angle brackets with your values. Two ways to point the command at the server, both supported:

From a checkout (no install step, PYTHONPATH points at src/):

{  "mcpServers": {    "modelmanager": {      "command": "/path/to/mcp-modelmanager/.venv/bin/python",      "args": ["-m", "mcp_modelmanager.server"],      "cwd": "/path/to/mcp-modelmanager",      "env": {        "PYTHONPATH": "src",        "MM_ACCESS": "direct",        "MM_VM_HOST": "<your-machine-or-127.0.0.1>",        "MM_VM_USER": "<your-ssh-user>",        "MM_CONTAINER_ROOT": "/srv/models"      }    }  }}

After pip install . (the console entry point, no cwd/PYTHONPATH needed):

{  "mcpServers": {    "modelmanager": {      "command": "/path/to/mcp-modelmanager/.venv/bin/mcp-modelmanager",      "env": {        "MM_ACCESS": "direct",        "MM_VM_HOST": "<your-machine-or-127.0.0.1>",        "MM_VM_USER": "<your-ssh-user>",        "MM_CONTAINER_ROOT": "/srv/models"      }    }  }}

Both blocks follow the plain mcpServers JSON shape most MCP clients read (Claude Code, Claude Desktop and any other MCP client that reads an mcpServers JSON block); consult your client's docs for where this block goes.

MM_ACCESS is direct (the server talks to the service ports under the given address) or tunnel (it talks to 127.0.0.1 and builds an SSH tunnel on demand; only sensible after the isolation, see SETUP.md). The default is tunnel; the example above sets direct because that is what a case a)/b) setup without the level-2 tunnel needs, so set MM_ACCESS=direct explicitly until that tunnel is in place.

Security model: fixed operations

There is no tool that runs an arbitrary command. Every operation is a fixed template in operations.py. The caller can only fill named placeholders, and every value must first pass an allowlist (validation.py): permitted is what explicitly matches the agreed form. It is not filtered out, it is allowed in. On substitution it is additionally quoted with shlex.quote; that is the net below the net, not the safeguard.

Checked on both sides: here before sending and, independently again, by a counterpart running on the model machine, which is bound to the control-channel key. So even if that key is stolen, only the closed set of operations is reachable.

Also: only targets on the own computer or the local network, names are resolved and re-checked at each use (no address pinning like the query server; prefer setting MM_VM_HOST to an IP if you use a hostname). No credentials in the code, no .env. The two SSH keys live outside the repo in your ~/.ssh; the .gitignore additionally excludes key patterns.

How the secured two-channel architecture (data channel for the tunnel, control channel for management) is set up and verified is described step by step in SETUP.md.

Operation tiers and the operator gate

Every tool has a fixed classification, kept in one place in the code (TOOL_TIER) so this table and the policy cannot drift apart:

This classification is not an authorization system (see the note in SECURITY.md); it is a coarse operator switch so you can narrow which operations this server will actually run, without editing code. A disabled operation stays listed over MCP but is refused on call, before any machine access:

Default is read-only (deny-by-default). Set nothing and only the READ tools run; CONTROL and DESTRUCTIVE operations must be switched on explicitly. The gate refuses a disabled operation early and cleanly, before any SSH or VM access, with a message that names the tier and how to change it. To also allow the management operations (fetch_model, switch_service, training, ...):

"env": { "MM_MAX_TIER": "control" }

Or expose exactly two operations and nothing else, note that MM_ENABLED_OPERATIONS narrows further on top of MM_MAX_TIER, so the tier must already admit both named tools, here switch_service is CONTROL, so MM_MAX_TIER must be raised to at least control:

"env": { "MM_MAX_TIER": "control", "MM_ENABLED_OPERATIONS": "status,switch_service" }

Concurrency guard

The machine has one GPU and one inference service, so two state changes at once would leave it inconsistent. Every GPU-state-changing tool runs its critical section serialized (fetch_model and cancel_training are excepted by design: fetch is a background job, and cancel must always get through): while one runs, a second is refused immediately (not queued), and the GPU-competing operations (switch_service, remove_model, create_variant, start_training, start_service) are refused while a training run this server started is still active. The guard is in-process: it makes this server's own calls consistent. A training run started from a different session is caught on the machine side (a new start_training stops inference first), not fully serialized here; that boundary is stated in SECURITY.md.

The tools

Constraints of a machine with one GPU

• Only one large model at a time. switch_service computes beforehand whether weights plus key-value cache plus base load fit into the allotted memory, and rejects rather than trying. Computed with the real file sizes and the model's config.json, not with a rule of thumb from the name; only if both are missing is there a rough estimate, then explicitly marked as uncertain.
• switch_service may only start a known container image and mount a known path. The container it (re)creates is checked against a closed image allowlist (the configured vLLM image MM_VLLM_IMAGE, default vllm/vllm-openai, plus ollama/ollama, optionally with a tag or digest; a mirrored vLLM image must be set via MM_VLLM_IMAGE or it is rejected) and every bind mount's host side must lie under one of the configured data roots; see SECURITY.md for the full reasoning.
• Inference and training do not share the card. start_training rejects as long as the confirmation to stop inference is not set, and names what would be stopped. Its numeric hyper-parameters (epochs, rank, alpha, learning rate, max length, batch size, accumulation) each have a fixed range that is checked on both sides: here before the run is written, and independently again on the machine before any value reaches the training library.
• The machine does not always have the card. If it is unreachable, the server reports that as a state of its own with an explanation, not as an error. If a second machine sharing the GPU is known via MM_VM2_HOST, the state is gpu_probably_on_second_machine, otherwise simply machine_unreachable.
• A switch that fails silently is the most dangerous failure mode. After every switch it is checked via /health and /v1/models whether the service really answers and serves the desired model. The result is always in the report.

Custom Ollama variants

Ollama lets you derive a model with a fixed system prompt, its own parameters and later a trained adapter, via a Modelfile. The derivation then appears as its own model name and is addressable like any other model. That is the path to specialized agents and, at the same time, the path by which training results enter operation.

create_variant(    name          = "handyman:v1",    base_model    = "qwen3:4b",    system_prompt = "You are a terse expert assistant ...",    parameters    = {"num_ctx": 8192, "temperature": 0.2})

The delicate spot: free text that runs into a directive file

A system prompt is prose the user must be able to determine. But it lands in a file Ollama reads as a directive list. Passing it through unchecked would bypass the whole protection, not over the shell but a layer above it, in the Modelfile itself. Four measures interlock:

1. The Modelfile is assembled on the machine, not transferred. Only parts go over the control channel: base model, an object of parameter name and number, an adapter path and the system prompt. There is no way to submit a finished Modelfile.
2. The permitted directives are in a closed list. The template can produce FROM, PARAMETER, ADAPTER and SYSTEM and nothing else. FROM takes an Ollama model name and no path and is checked against the actual inventory.
3. The free text travels as Base64. No shell syntax and no newline can arise in transit from the Base64 alphabet; it is unpacked only on the machine, straight into the file.
4. The character set of the free text contains no double quote. A multi-line Modelfile value ends with three of them; with the character missing from the set, the text cannot end its own block and never become a directive. For quotes the typographic quotation marks and the apostrophe are available.

Checked on both sides, since validation.py runs here as on the machine. A crafted request placed straight into the control channel, past the check here, is rejected by the wrapper on its own.

The tunable parameters

Mapped are the ones actually worth using, not everything Ollama knows. Each has a range that is checked before the value reaches the machine.

Adapter: prepared but not live

The adapter parameter is built and checked, but the path has never run, because there is no trained adapter in the reference state. What is present: the allowlist for the adapter path, the conversion to the container's view, the ADAPTER line in the template and the reading in the display tools. What has to be settled with the first real adapter (Safetensors vs. GGUF format, the storage location in the container, the fit to base model and quantization) is noted as open in the code, in place. Until that is proven: leave adapter empty.

Selftest

.venv/bin/python selftest.py          # without the machine, HTTP is mocked.venv/bin/python selftest.py --live   # additionally against the real machine

Most groups run without a machine; only the optional --live group M needs one. The operator-side setup scripts (setup/, vm_side/) that some checks exercise ship with this repo, next to the server, so with them in place the selftest runs additional checks on this machine, no model machine required: on top of the server-code groups they cover the wrapper in actual execution (group D), the setup script itself (group J), the isolation (group K) and the operator policy, the concurrency guard and the isolation helpers (group N). Allowlists against an attack battery, catalog and substitution, request format, the service side against a stub server, memory estimation against hand-checked values, connection layer, parsing of the machine outputs and the variants all run and are checked. Should you run the server directory on its own, with the setup/ and vm_side/ scripts removed from next to it, the selftest still passes the server-code groups and skips groups D, J, K and the operator-only parts of I and N cleanly with a note, not counted as failed. Neither scenario needs a model machine. Only the --live group M does: it runs against a reachable, set-up machine and is optional on top of either scenario. (The exact check count is what the runner prints on the run in front of you; it grows as checks are added, so it is not quoted here as a fixed number and is not contractual.)

--live does NOT read MM_VM_HOST/MM_VM_USER/MM_CONTAINER_ROOT; it uses its own, separate variables, defaulting to an address that is never reachable (TEST-NET, RFC 5737) so a plain --live run without further setup is silently skipped rather than a false pass:

Set all three to point --live at your actual, reachable machine; otherwise you will see "VM not reachable, live group is skipped. Not an error." even though MM_VM_HOST itself is configured correctly for normal server use.

Environment variables

Required fields are marked required; they have no default value.

The local tunnel ports default to the same well-known values as the remote ports; adjust them if those are already taken locally.

MM_CONTAINER_ROOT is required and has no built-in default (there is no default machine layout this tool could assume); without it the server exits at startup. The six MM_VM_*/MM_LORA_PYTHON variables above derive from it by default and stay individually overridable, exactly like MM_IN_CONTAINER_STORE, so a machine with a DIFFERENT directory layout can be adapted through configuration alone, no code change needed. Every value is checked at startup against a strict allowlist for VM paths (absolute, letters/digits/dot/hyphen/ underscore/slash only, no .., no spaces, no shell characters); a malformed value stops the server immediately instead of being trusted. MM_VM_DATA_ROOT and MM_VM_OLLAMA_STORE additionally become the allowlist roots for every path a tool call may pass in later (a training dataset path, an adapter path), so they stay consistent with what is actually configured.

Places on the machine that a variant needs, derived from the roots above (in operations.py, not separately settable). The setup creates these paths:

License

MIT, see LICENSE.