Hone

The 30-second version

Every AI eval tool answers one question: was the model's output good? (accuracy, hallucination, toxicity, latency, cost.)

None of them answer the question that actually decides whether AI-assisted work is safe to ship: does the human who produced it understand it well enough to defend it?

Hone is an open toolkit for measuring exactly that. Give it a piece of AI-assisted work; it interviews the human — a few sharp questions, one at a time, the way a good mentor would — then returns their AI-Q: a score over the three Ds of working with AI. Discernment (can you tell when the AI is wrong? — six auditable dimensions, the spine of the score), Delegation (did you brief it, choose the task, and steer it well?), and Design (are you re-architecting how you work? — read over time, never faked from one review).

It's bring-your-own-model (any LLM), Apache-2.0, and drops into a coding agent, a CI pipeline, or your own app in minutes. The methodology is an open spec — the AI-Q — the open standard — so the score means the same thing everywhere it's computed.

What a review feels like

You:  Run hone on the pricing analysis we just built.
Sol:  Happy to take a look. This is a strategy memo recommending a Pro-tier      price increase — real money riding on it, so I'll ask five questions.      Just between us, and "I don't know" is a perfectly useful answer.
      First: walk me through the recommendation in your own words —      why $49, and not $39 or $59?
      ...three more, one at a time — including: what did you give the      AI to work with, and what did you have to correct in its first      answer? ...
      Last one: before I show you anything — honestly, how confident      are you in this work, 0 to 100?
You:  85.
Sol:  Here's my read.
      Pro-tier pricing recommendation · AI-Q 58 — Mostly owns it      (discernment 57 · delegation 60)      You can explain the mechanism, but you took the churn projection      on faith — and that's the load-bearing number.
      Work on this first: the 4% churn assumption came straight from the      AI with no source. If churn runs 8%, the revenue case inverts.
      [ what a rigorous review surfaced vs. what you caught ]      [ before-you-ship checklist · the toughest objection you'll face ]      [ worth remembering · 3 short notes on the skills you missed ]

No grades for the work. No "AI detector." A measure of your grip on it — and coaching to close the gap.

Quickstart

In your coding agent (the best way) — review work where it lives

When an agent built the thing — a repo, an analysis, a doc set — don't paste it anywhere. Hone runs inside the agent, which already has the work loaded:

Claude Desktop — one click: download hone.mcpb from the latest release and open it (Settings → Extensions). No config files, no terminal.

Claude Code — add the MCP server (protocol + deterministic scoring):

git clone https://github.com/derekchoyai/hone && cd hone/mcp && npm install && npm run buildclaude mcp add hone -- node "$(pwd)/dist/index.js"

Or load the Claude skill — zero infrastructure. Then just say:

"Run hone on what we built today."

The agent reads the work it already has, interviews you one question at a time, commits your confidence before the reveal, and computes your AI-Q deterministically. With your okay it remembers the scores — never the work — in a local file, so the next review pushes on your actual weak spots ("your verification has been the soft dimension — let's start there"). Nothing leaves your machine.

Gate AI-generated code in CI (GitHub Action)

# .github/workflows/judgment.yml- uses: derekchoyai/hone/integrations/github-action@v0  with:    domain: coding    paths: "src/**/*.ts"    model: openai            # or anthropic  env:    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

On a PR touching AI-generated code, the author answers a few questions and the work gets a Judgment Score. Below your threshold → the check explains the gaps. See integrations/github-action.

Build it into your own app (TypeScript SDK)

npm install hone-sdk

import { analyzeWork, reviewJudgment, type ModelFn } from "hone-sdk";
// Bring your own model — Hone never sees your API keys.const model: ModelFn = async ({ system, user }) => callYourLLM(system, user);
// 1. Decompose the work + generate an interview tailored to it.const { workMap, questions } = await analyzeWork({  work: aiGeneratedArtifact,  domain: "coding",  model,});
// 2. Collect the human's answers (in YOUR ui), then score.const result = await reviewJudgment({  work: aiGeneratedArtifact,  domain: "coding",  answers,        // [{ question, answer }]  model,});
console.log(result.composite, result.band);       // 72  "Mostly owns it"  ← the AI-Qconsole.log(result.discernment, result.delegation); // sub-scores (delegation only when assessed)console.log(result.gaps, result.verifyBeforeShip);

result conforms to spec/score.schema.json.

No install at all — run the prompts by hand

Paste a prompt into any chat model with your work. See examples/ for fully worked reviews with scores.

AI-Q — the score

A transparent 0–100 score derived from a rubric you can read and change — never vibes, never model arithmetic. AI-Q measures the three Ds of working with AI, weighted by how much they protect you (Discernment 0.6 · Delegation 0.3 · Design 0.1, renormalized over what was actually assessed):

D1 · Discernment — judgment about the AI's output (the spine)

Can you tell when the AI is wrong, lazy, or hallucinating? Six auditable dimensions, always assessable from the work in front of the reviewer:

D2 · Delegation — judgment about your input to the AI

Did the brief carry what the task needed? Was this the right thing to hand to AI at all? Did you steer between the first output and the final one? Three facets — brief quality, task selection, iteration control — scored only when the interview actually surfaced the brief (the evidence gate). No evidence → reported as not assessed, never guessed, never zero.

D3 · Design — judgment about your system of work

Are you re-architecting how you work around AI — or just using it where it lands? One artifact can't show a system, so Design is never scored from a single review. It's read qualitatively from your profile over time (domain breadth, delegation trend, whether discernment holds as you delegate more); a numeric Design score needs cross-user baselines and stays on the roadmap.

Agentic work, too. When the thing you made is a system — an agent loop, an automation, a multi-agent setup — the same three Ds re-point to the loop: is the "done" check real or just "looks done" (Discernment), was a loop even the right call vs. a one-shot prompt and is the blast radius bounded (Delegation). Hone measures judgment over how you run AI, not just one-shot answers. See spec/aiq.md.

Bands: 80–100 Owns it · 55–79 Mostly owns it · 30–54 Riding the AI · 0–29 Black box.

Scoring is deterministic — every sub-score and the composite are computed by the same code everywhere (hone-sdk), so a 72 in CI means the same thing as a 72 in your agent. A six-dimension review's AI-Q equals its Discernment sub-score, so every v0.1 Judgment Score is already a valid AI-Q. The full normative definition is spec/aiq.md; two extended Discernment dimensions — Counterargument and Curiosity — are optional and defined in spec/judgment-dimensions.md.

A score of judgment has to itself be defensible. Each dimension is scored independently against published 1–5 anchors (not a holistic vibe), with explicit guards against the self-preference and fluency biases an LLM judge is prone to, and the implementation leads with the band, not a false-precision integer — with an ensemble option for high-stakes reviews. The full rules are in spec/aiq.md → Scoring rigor.

The interview methodology is built on the evidence, not vibes either: premortems (Klein), consider-the-opposite and reference-class forecasting (Tetlock/GJP), calibration against stated confidence (Kahneman–Klein), and confidence-before-reveal — because subjective confidence is not an accuracy signal.

Why AI evaluation isn't enough

Current AI evaluation asks: was the model correct? Hone asks: does the human understand?

As AI collapses the cost of producing work, the scarce skill becomes recognizing good work — knowing what to trust, what to verify, and what you're prepared to defend with your name on it. A 2025 CHI study of 319 knowledge workers (Lee et al., Carnegie Mellon & Microsoft Research) found confidence in AI associated with less self-reported critical thinking (β = −0.69, p < 0.001), while confidence in one's own ability was associated with more (β = +0.26). That gap — between what the work needed and what the human caught — is the judgment gap, and nothing in the eval stack measures it.

The un-gameable layer. Automated metrics get gamed: in 2026, a single automated agent was shown to break every major agent benchmark via reward-hacking — near-perfect scores without solving the task. The one thing that can't be optimized away by the model is whether the human can stand behind the result. That's what Hone measures, and why it scores the person rather than the output.

Full argument: The AI Judgment Gap — Why AI Evaluation Is Not Enough.

How it works (the architecture bet)

The host does the reading; Hone does the judging. When your agent already has the repo/docs loaded, re-ingesting them into a review service duplicates the hardest infrastructure in the industry — and leaks your work. So Hone inverts it: methodology flows in (rubrics, interview protocol, question altitude), judgment metadata flows out (six scores, a band). The work itself never transits. Privacy isn't a policy here; it's the architecture.

┌─ your agent / CI / app ──────────────────────────┐│  has the work loaded · runs the interview        ││                                                  ││   get_protocol ──►  AI-Q spec: rubric, questions, ││                     techniques, report structure ││   score_review ──►  deterministic AI-Q:          ││                     sub-scores + composite+band  │└──────────────────────────────────────────────────┘        nothing stored · nothing leaves the machine

Repo map

spec/             the AI-Q standard — dimensions, JSON schemas, agentic-integration specrubrics/          per-domain review rubrics (YAML, editable)prompts/          copy-paste prompt library (works in any chat model)examples/         fully worked reviews with scorescalibration/      human-scored gold transcripts + judge-calibration harnesssdk/typescript/   the drop-in, bring-your-own-model client        ← hone-sdkmcp/              MCP server — judgment review inside any agent   ← start hereskills/           the "hone" agent skill (Claude)integrations/     github-action + cliWHITEPAPER.md     the argument

Status & roadmap

v0.1 is an early, honest draft — a framework, not "the standard." Standards are earned through use. The highest-value contributions right now: rubrics for new domains, worked examples, language ports of the SDK, and integrations. See CONTRIBUTING.md and spec/roadmap.md.

Open core

Everything here is and will remain open (Apache-2.0): the AI-Q spec, rubrics, prompts, schemas, SDK, MCP server, and integrations. A hosted product — Hone, the app (in private beta) — builds on this framework with the parts that need scale and data: longitudinal AI-Q profiles, adaptive coaching that learns your weak spots, spaced recall, cohort benchmarks, and the numeric Design score (D3) that needs cross-user baselines. The framework stays free forever.

License

Apache-2.0. Use it commercially, embed it, fork it.