Local LLM Guard for Ollama / llama.cpp

AI Guardian

Audit availability: Real writes require a committed pending audit row and stop before contacting the platform if storage is unavailable. Reads and previews may continue with a warning; new audit rows store outcome summaries rather than full responses.

Disclaimer: Community-maintained open-source project. Not affiliated with, endorsed by, or sponsored by Ollama, IGEL, or any AI-security vendor. Product and trademark names belong to their owners. MIT licensed.

Governed observability + governance for on-endpoint local LLMs. It lets you observe + audit what your local models are actually fed, and gate what leaves in a prompt — the complement to IGEL AI Armor. AI Armor governs whether a local model may run on the endpoint; ai-guardian records what it did and gates what goes into the prompt (secrets, PII, source, jailbreaks) plus which model may serve it. Self-contained: it talks to each runtime's REST API and needs nothing beyond httpx and the MCP SDK. v0.1 provides opt-in route-through content governance, plus a transparent capture proxy for clients that did not opt in.

Supported runtimes

One tool, several local runtimes, selected per target by a runtime field in config.yaml (the init wizard asks). Ollama uses its native API; the other three share one OpenAI-compatible transport (/v1/models + /v1/chat/completions).

The allow/deny model policy, the deterministic prompt scanner, the route-through guard (guarded_generate / observe_chat), provenance drift, and doctor work across all runtimes. Model lifecycle writes (pull / remove / unload) are Ollama-only — the OpenAI-compatible servers load a model at startup and expose no lifecycle endpoint, so those writes are refused with a clear message.

Provenance honesty: only Ollama (content digest) and llama.cpp (a /props-derived path/size identity) expose something to pin. LM Studio and vLLM expose only a model id, so a pinned digest is reported unverifiable rather than a false DRIFT.

vLLM here is a LOCAL endpoint-guarding use case. GPU inference-cluster operations (autoscale, drain, Ray Serve/Jobs, model lifecycle at fleet scale) belong to a different tool in the line — GPU cluster ops → inference-aiops.

What it does

Ollama persists no queryable prompt/response history — conversational context is client-supplied on every request. So ai-guardian observes on two fronts:

• Passive inventory / state auditing — over /api/tags, /api/ps, /api/show, /api/version: what models are installed and running, their VRAM residency, license/params/capabilities, and their provenance digests. Every model is annotated with an allow/deny policy verdict, so shadow (unsanctioned) models show allowed: false.
• Opt-in route-through content governance — callers send a prompt through ai-guardian (guarded_generate / observe_chat). It scans the text (secrets / PII / source / jailbreak), checks the model against policy, records the interaction to its own usage log (~/.ai-guardian/usage.db), and only then calls Ollama — blocking when the risk band is too high or the model is disallowed. The raw prompt is never stored (only its length + redacted findings).

Now available (ai-guardian proxy serve): a transparent capture proxy that applies the same scan + model policy + recording to traffic from clients that never opted in — point them at the proxy instead of the runtime. It inspects requests and streams responses through untouched, and it is a chokepoint, not an enforcement boundary: a client that can still reach the runtime's real port bypasses it entirely. proxy_guidance returns the command and that caveat; the CLI prints it on every start.

Key features

• Deterministic, offline prompt scanner — no I/O, no network, so it is fully testable offline. Flags secrets (AWS AKIA, private-key blocks, GitHub / Slack / OpenAI / Google tokens, JWTs, assigned api_key=…, high-entropy fallback), PII (email, US SSN, credit card with a Luhn check), source/config-leak heuristics, and jailbreak / prompt-injection signatures — rolled up into a weighted risk band (low / medium / high / critical; any critical dominates). Findings are redacted — the scanner never re-emits the secret it caught.
• Model allow/deny policy (shell-glob patterns) so shadow / unsanctioned models surface as allowed: false, plus provenance digest pinning to flag a model whose digest drifted (re-pulled / tampered).
• Route-through guard — guarded_generate / observe_chat scan + policy-gate
  • record + run-if-allowed, blocking on risk-band >= block_threshold (default high) or a disallowed model.
• Vendored governance harness — audit log, token/runaway budget guard, descriptive risk tiers, and undo-token recording, bundled in the package (no external dependency).
• Highly self-testable — Ollama is free + local for the API parts; the scanner, policy, and risk-band are pure deterministic offline logic.

What this tool does, and does not, decide

It delivers local-LLM observability and operations — reads and writes — accurately, and records every one of them. It does not decide whether a write to the model estate is allowed to happen. That is the agent's judgement, or the permission of the host and account you run it under: point it at a runtime the account cannot administer — an Ollama daemon whose model store the user can't modify, or an endpoint the agent reaches read-only — and the writes fail at the runtime, the place that actually owns the permission. Simplest of all, hand the connecting agent only the scan/observe tools.

So the harness has no read-only switch, no deny-rules file, and no approval gate to configure. (Content governance is a separate, product-level thing that stays: the model allow/deny policy and the guarded_generate block threshold still scan and gate what a model is asked to do.) The one thing the harness guarantees is that nothing is silent: every call, over MCP and over the CLI alike, lands an audit row in ~/.ai-guardian/audit.db, and destructive writes still capture their before-state and record an inverse where one exists.

Each tool declares a risk_level, kept in agreement with its [READ]/[WRITE] documentation tag by a test, and carried into the audit row as a descriptive tier — so a reviewer can see at a glance that a row was a high-risk delete. It is a label, not a gate.

Running a smaller / local model? See agent-guardrails.md — it lists the guardrails this tool now enforces for you (so you don't spend prompt budget restating them) and gives a ready-made system prompt for what's left.

Capability matrix (21 MCP tools)

Reads (11)

Writes (8)

Undo (2)

Risk-band gating: guarded_generate / observe_chat block when the prompt's risk band >= block_threshold (default high) or the model is disallowed. Blocked calls never reach Ollama and are recorded as blocked in the usage log.

Quick start

As a Claude Code plugin

One install gives an agent both the skill and the MCP server:

/plugin marketplace add AIops-tools/marketplace/plugin install ai-guardian@aiops-tools

The MCP server is fetched with uv and pinned to the package version this plugin declares, so an audit row can be traced back to the code that wrote it. Credentials are still configured with ai-guardian init — see below.

As an OpenClaw plugin

The same bundle is published on ClawHub, where one install delivers the skill and its MCP server together:

openclaw plugins install clawhub:@zw008/ai-guardianopenclaw skills info ai-guardian          # expect: Visible to model: yes

Restart the OpenClaw gateway afterwards so it loads the plugin. The MCP server is fetched with uv, pinned to this exact release, so uvx has to be on PATH — without it the skill still installs but reports Visible to model: no. Credentials are configured exactly as below.

As a CLI or standalone MCP server

uv tool install ai-guardian-aiops          # or: pipx install ai-guardian-aiopsai-guardian doctor                   # Ollama reachability + policy summary (works zero-config)ai-guardian overview                 # models installed/running, shadow count, usage statsai-guardian model list               # installed models with allow/deny verdictsai-guardian guard scan "my key is AKIAIOSFODNN7EXAMPLE"   # deterministic scan → risk band

The command and the package have different names. The CLI, the skill and this repo are all ai-guardian; the PyPI distribution is ai-guardian-aiops because ai-guardian was already taken. Both spellings below are correct and intentional — install ai-guardian-aiops, then run ai-guardian.

Route a prompt through the guard (scan + policy-gate + record + run-if-allowed) via MCP:

guarded_generate(model="llama3.2:3b", prompt="…", block_threshold="high")

Run as an MCP server (stdio) — the full 21-tool surface; the CLI is a convenience subset:

export AI_GUARDIAN_AIOPS_MASTER_PASSWORD=...   # only if a target has a stored tokenai-guardian mcp                                # or: ai-guardian-mcp

Where that password then lives: an exported variable is readable by every process this shell starts and is recorded by shell history. On a shared or long-lived host, prefer the interactive prompt, or inject it from a secret manager for the life of the one command that needs it.

Governance

Every operation — MCP and CLI — passes through the bundled @governed_tool harness. It records; it does not authorize (see above).

• Audit — every call (params, outcome summary, status, duration, risk tier, and any operator-supplied approver/rationale) is logged to ~/.ai-guardian/audit.db (relocatable via AI_GUARDIAN_AIOPS_HOME). This is separate from ~/.ai-guardian/usage.db, which holds the observed local-LLM usage.
• Runaway guard — a safety backstop, not an authorization gate: the same call hammered in a tight loop trips a circuit breaker so a stuck agent can't burn unbounded calls/time. Disable with AI_GUARDIAN_RUNAWAY_MAX=0; optional hard ceilings via AI_GUARDIAN_MAX_TOOL_CALLS / AI_GUARDIAN_MAX_TOOL_SECONDS.
• Undo recording — reversible writes record an inverse descriptor built from the fetched before-state.
• Risk tier — a descriptive label on the audit row derived from risk_level; it gates nothing.

Supported scope + limitations

• Scope: on-endpoint local LLMs — Ollama plus the OpenAI-compatible llama.cpp / LM Studio / local single-node vLLM — single-endpoint local-LLM observability + content governance. Not GPU inference-cluster ops (→ inference-aiops).
• v0.1 = passive inventory/state auditing plus opt-in route-through content governance. A transparent capture proxy for other clients' traffic is v0.2 roadmap, not v0.1.
• IGEL AI Armor interop is doc-level positioning today (complementary roles), not a wired integration.
• Validation status — the scanner, policy, and risk-band are pure deterministic offline logic and are exercised as such by the test suite. The core Ollama route-through (real generation + policy deny + undo capture) was exercised against a live Ollama 0.24.0 on 2026-07-13; the rest of the Ollama surface and the OpenAI-compatible dialects (llama.cpp / LM Studio / local vLLM) are still covered by mocked responses only. ai-guardian doctor is the fastest live check; see docs/VERIFICATION.md for exactly which boxes are ticked.

Missing a capability?

Want a passive capture proxy, another scanner signature, a richer policy model, or an AI Armor hook? Open an issue or PR — feedback and contributions welcome.