Quellgeist Metrics

Quellgeist

[ci] [security] [tuned 4B] [License: MIT] [Python 3.12+]

First-line incident triage you can trust: ranked root-cause hypotheses where every claim cites a real evidence handle — and the agent abstains rather than guess.

Quellgeist is a model-agnostic AI agent for first-line production-incident triage. It runs a legible JSON-action ReAct loop over read-only tools (structured logs + recent deploys + metric time-series), then emits a structured Diagnosis: confidence-ranked root-cause hypotheses, each backed by a structured evidence handle (LogRef.id / CommitRef.sha / MetricRef.id) the agent actually saw — never free text. Two ideas set it apart:

• Cite-by-structured-handle. Evidence is a checkable handle, not a sentence, so a fabricated citation is measurable and deterministically rejected by a keyless fabrication check — not a matter of fuzzy string-matching.
• Abstain-over-hallucinate. A confidently-stated wrong cause is the worst possible answer, so "insufficient evidence" is a first-class outcome.

Status: Wave 4 complete — the fine-tune works. The DR-0020 QLoRA fine-tune of the local reasoner (Qwen3-4B, served via Ollama) took it from the base's 0/16 holdout to 12/16 — zero fabrication, zero speculative-filtering, and cheaper than the base — while beating a 31B frontier (Gemma-4-31B, 10/16) on the same holdout at $0, fully offline. Non-memorisation is triangulated three ways (fixtures ≈ holdout; core-fresh ≥ core-overlap; structure probe 7/10). Two honest limits: the resource_exhaustion class didn't transfer (0/N; the frontier passes it), and adversarial-abstention recall is 6/12 at the system level — a ceiling the 31B frontier shares (also 6/12), not a fine-tune regression. When this agent misses it's incomplete or too cautious, never confidently fabricating. See Status & roadmap · fine-tune case study.

Why it's different

What it is / what it's NOT

• It is: a first-line triage agent — ranked, evidence-cited root-cause hypotheses (or an honest abstention) from read-only logs/deploys/metrics, over a model-agnostic loop that runs on a hosted frontier model or a local 4B.
• It is NOT: an autonomous remediator (it never mutates prod — resolution verification is a deferred, cut-first wave); a production-hardened service (the demo is a deliberate toy); or a general-purpose agent. The holdout it's measured on is out-of-vocabulary but in-structure — not a claim about unseen incident shapes or real production data.

Quickstart (~30 seconds to a broken service + structured logs)

Requires uv and Python 3.12+.

See a real-shaped diagnosis in one keyless command (no model, no API key):

uv sync && uv run quellgeist diagnose --demo     # renders the demo incident's cited postmortem

Then run the full loop against the live toy service:

uv run uvicorn demo.app.main:app          # 1. start the toy service (leave running)
# --- in a second shell, from the repo root ---uv run python -m demo.chaos.bad_deploy    # 2. inject a simulated bad deploycurl -s localhost:8000/login              # 3. trip /login -> 500s + structured error logsuv run quellgeist diagnose --show-trace   # 4. diagnose live (needs a model; see below)
uv run python -m demo.chaos.reset         # back to a green slate

The live step needs a reasoner — see Running the model. Without a key, quellgeist diagnose exits 1 with a one-line error + hint (never a traceback); --demo always works keyless and renders the same output shape deterministically from gold.

Architecture

A custom, legible loop is the orchestration layer; the three read-only tools are the evidence interface; the Diagnosis schema is the contract that the postmortem renderer and the eval judge both read.

flowchart TD    trigger(["incident trigger - CLI"]) --> loop    model["reasoner via LiteLLM<br/>(Gemini or local Qwen, swappable)"] -. "chat completion" .-> loop
    subgraph loopbox["model-agnostic JSON-action ReAct loop"]      loop["run_loop()<br/>decide, call tool, observe, repeat"]    end
    loop -- "query_logs" --> logs["logs tool<br/>structured JSONL, stable ids"]    loop -- "get_recent_commits" --> commits["commits tool<br/>deploy_log.json, shas"]    loop -- "query_metrics" --> metrics["metrics tool<br/>time-series, named series"]    logs -- "rows + ids" --> loop    commits -- "commits + shas" --> loop    metrics -- "series + names" --> loop
    loop --> diag["Diagnosis (schema.py)<br/>ranked hypotheses citing<br/>LogRef.id / CommitRef.sha / MetricRef.id, or abstains"]    diag --> pm["postmortem renderer<br/>deterministic Markdown"]    diag --> judge["eval judge<br/>fixture scenarios, CI gate"]

All three tools are also exposed as MCP servers over stdio (python -m quellgeist.servers.logs_mcp, …commits_mcp, …metrics_mcp). The agent currently reuses the same tool functions in-process behind a ToolSpec registry; a stdio MCP-client path (the agent driving the servers over the wire) is on the roadmap (DR-0010).

Deep dive: docs/architecture.md walks the full pipeline (loop → tools → verifier → postmortem), a sequence diagram, the module map, and the cross-cutting design decisions.

The servers publish to the Official MCP Registry on each tagged release (see docs/publishing.md); once published each is runnable with uvx --from quellgeist quellgeist-logs-mcp (or …-commits-mcp / …-metrics-mcp).

Example session

Inject the bad deploy — it drops a marker that flips verify_token into a NoneType regression and writes a deploy_log.json whose offending commit landed just before the errors (illustrative stdout — the timestamp reflects when you run it; paths shown relative to the repo root):

$ uv run python -m demo.chaos.bad_deployinjected bad deploy a1b2c3d (touched demo/app/auth.py) at 2026-06-24T12:22:43Z  marker:     demo/.bad_deploy  deploy log: demo/deploy_log.jsonnext: hit /login to generate the 500s, then `quellgeist diagnose`

With a reasoner configured, quellgeist diagnose reads the logs + deploys and emits a postmortem. The CI environment has no validated model key (DR-0012), so the diagnosis below is rendered from gold — built deterministically from the fixture's labelled cause and evidence handles via render_postmortem, not live model output:

# Incident Postmortem (rendered from gold)
## Root-cause hypotheses
### 1. Bad deploy a1b2c3d (10:01:50Z) refactored auth.py and introduced a NoneType error in verify_token; /login 500s begin ~20s later at 10:02:12Z.  (confidence: 1.00)
Evidence:- log #2- commit a1b2c3d

Reproduce that render yourself (no model needed):

uv run python - <<'PY'from evals.scenarios.generator import load_scenariofrom quellgeist.agent.schema import Diagnosis, Hypothesisfrom quellgeist.output.postmortem import render_postmortem
s = load_scenario("evals/scenarios/fixtures/bad_deploy_0001.json")gold = Diagnosis(hypotheses=[    Hypothesis(cause=s.gold_cause, confidence=1.0, evidence=s.gold_evidence_refs)])print(render_postmortem(gold, title="Incident Postmortem (rendered from gold)"))PY

The point isn't the prose — it's that log #2 and commit a1b2c3d are exact handles into the real signals, not paraphrases. A live run additionally fills in a one-line summary and suggested actions, and abstains outright when the evidence is too weak to name a confident cause.

Write the postmortem to a file with --out postmortem.md, or as a self-contained HTML page with --out postmortem.html (or --format html) — same deterministic render, no external assets.

Running the model

The reasoner is any LiteLLM model string, selected by --model or the QG_MODEL env var (default gemini/gemini-3.5-flash). Provider keys are read from the environment by LiteLLM; nothing is stored in the repo.

export QG_MODEL="gemini/gemini-3.5-flash"export GEMINI_API_KEY="…"uv run quellgeist diagnose --show-trace

Or fully local and offline via Ollama — the intended home default (DR-0008; exact artifact pinned in DR-0019), no API key involved:

ollama pull qwen3:4b-instruct-2507-q4_K_Mexport QG_MODEL="ollama_chat/qwen3:4b-instruct-2507-q4_K_M"uv run quellgeist diagnose --show-trace

Base vs tuned — important. The ollama pull above is the base Qwen3-4B: the honest safe floor — it scores 0/16 on the holdout and abstains on everything, never fabricating (DR-0019). The 12/16 headline is the DR-0020 fine-tune (quellgeist-qwen3-dr0020), which you build + serve via finetune/README.md (a free-Colab QLoRA run → ollama create). Until that tuned GGUF is published for a one-line pull, the base model is what a plain ollama pull gives you — safe, not yet useful. Use a hosted model (above) or the fine-tune to see live diagnoses.

Heads-up (DR-0012): a Gemini key on an unvalidated, no-billing project returns 429 limit: 0 on current models, so the shipped CI gate is deliberately keyless and model-driven evals are key-gated and run out-of-band (DR-0015). At home the intended default reasoner is a local Qwen3-4B via Ollama (DR-0008).

Running the eval (reasoner + verifier + LLM-judge)

The fixture eval scores the reasoner with a deterministic keyword judge + a zero-fabrication check (the keyless gate), and can additionally run two model layers (DR-0016): a verifier that confirms cited evidence supports each hypothesis (forcing abstention otherwise) and an advisory LLM-judge rubric.

export GEMINI_API_KEY="…"export QG_MODEL="gemini/gemini-3.5-flash"QG_VERIFY=1 QG_JUDGE_LLM=1 \QG_MIN_CALL_INTERVAL_S=6 \      # pace calls under the free-tier RPM (avoids 429 bursts)  uv run python -m evals.run_evals

QG_VERIFIER_MODEL / QG_JUDGE_MODEL override the model per layer (default QG_MODEL). An unreachable backend (quota/503/timeout) or a rejected credential (missing/invalid/stale key) is reported as a skip, not a failure (DR-0015/DR-0017), so the out-of-band eval never reddens on a free-tier hiccup. The LLM-judge's scores are advisory (they never gate). On a human-labelled gold subset it agreed with human verdicts at Cohen's kappa 0.81 using an independent judge (groq/llama-3.1-8b-instant ≠ the reasoner) — validated on that subset (DR-0018); still self-grading whenever QG_JUDGE_MODEL equals the reasoner.

CI's out-of-band eval runs on Groq (groq/llama-3.3-70b-versatile, gated on GROQ_API_KEY): Gemini's free tier proved unusable from cloud CI (429 → 503 → timeout → invalid-key), so the reasoner was swapped with one env var — the model-agnostic thesis in action (DR-0017). The intended home default remains a local Qwen3-4B (DR-0008).

Using it on your real data

The demo eats three canonical files; your production signals don't look like that. quellgeist ingest is the adapter — point it at real sources and it writes the canonical files the tools read:

quellgeist ingest \  --logs    /var/log/myapp/     # file or directory; JSONL, JSON, plain text, or mixed  --deploys deploys.json        # JSON array, GitHub payload, or `git log` text  --metrics prom.json           # a Prometheus response or a canonical array  --out-dir ./signals# prints the `export QG_*` lines; then:quellgeist diagnose --show-trace --strict-citations   # add --model / a provider key

It tolerates messy real data (foreign field names are aliased onto the schema, timestamps normalised to UTC, a malformed line coerced rather than crashing the run), and query_logs caps how many rows one observation returns (QG_MAX_ROWS, default 200) so a large log can't blow the context window. The deterministic cite-or-abstain guarantee runs at real-use time: diagnose verifies every cited handle against your real signals and warns on a fabrication (--strict-citations exits non-zero for CI). Full guide: docs/ingestion.md.

Run the live service (v2)

v2 wraps the same frozen core in a live, concurrent, observable incident-response service: a signed webhook triggers an investigation, a worker pool runs the unchanged loop over an isolated per-incident snapshot, every run is persisted to SQLite with its trace and cost, an operator approves / steers / rejects before it posts to Slack + a self-contained HTML page, and after a sandbox fix the agent re-reads signals to confirm recovery. Everything additive; the frozen measurement surface is untouched. The whole stack runs from one file:

cp .env.example .env            # set QG_WEBHOOK_SECRET, QG_OPERATOR_TOKEN, QG_VERIFIER_MODEL…docker compose up --build       # demo service + agent service + Ollamadocker compose exec ollama ollama pull qwen3:4b-instruct-2507-q4_K_M   # first run only
# break it, trigger an investigation, review, then confirm the fix in the sandbox:docker compose exec demo python -m demo.chaos.bad_deploy#   … POST a signed incident to :8000/incidents, approve it on the HTML page …docker compose exec demo python -m demo.chaos.fix_deploy   # heal without wiping the log#   … POST /incidents/{id}/verify-resolution → recovered | not_recovered | inconclusive

Secrets stay env-only (public repo); the service is fail-closed — no webhook secret rejects every request, no operator token closes the operator surface, and it never posts a fabricated or unverified diagnosis. Design: DR-0023 + spec.

Status & roadmap

Built in rolling waves — only the current wave is implemented in detail (see docs/quellgeist-plan-rolling-wave.md). The full decision history lives in the ADR log.

The wave boundary is deliberate, not unfinished: only the current wave is built in detail, and later waves are scoped but intentionally unimplemented. v2 is additive over the proven v1 core — the frozen fine-tune measurement surface is never touched (DR-0023; guarded by tests/frozen/).

Reliability gate

The deterministic CI gate is the reliability contract: 339 tests (ruff + black via pre-commit, then pytest — covering the loop's never-crash / graceful-abstention behaviour, the deterministic fabrication check and cite-based judge gate, the verifier and advisory LLM-judge, parameterised scenario generation, the judge-validation harness, the server filters, the postmortem renderer, the fixture-backed eval harness, the real-data ingestion + robustness layer, an end-to-end real-incident harness, and the v2 live service — signed webhook, concurrent per-incident isolation, HITL review gate, and the deterministic sandbox resolution check) on Python 3.12 and 3.13.

Out of band, the model-driven eval runs the reasoner over the 65-scenario suite. The latest full run scored 61/65 passed, 0 fabricated evidence (Cerebras Gemma-4-31B) — per-class breakdown + the failure analysis in the reliability case study.

uv run pytest tests/ -quv run pre-commit run --all-files

Development & contributing

See CONTRIBUTING.md for the dev setup, conventions, and the wave model; SECURITY.md for reporting and the no-secrets / toy-demo policy; and CODE_OF_CONDUCT.md for community expectations. Bug reports and feature requests use the issue templates; PRs follow the PR template.

License

MIT © Rajeev Shyam Kumar.