whetstone
Two learners grinding each other sharp: a student that must earn every
promotion, and an examiner that must earn the right to judge it — with every
label anchored to an oracle that cannot practically be wrong.
whetstone is a self-improving research harness built around one rule: nothing is
believed until something exact has tried to kill it. A verifier-gated student
(QLoRA adapters on a local 4B) trains on experience milled from root oracles —
exhaustive graph enumeration, simulated-annealing counterexample search, exact
game simulators, Monte-Carlo consensus, Stockfish, KataGo — and is graded by an
evolving exam bank that quarantines anything the student trained on, learns which
items discriminate, retires the saturated, and feeds its retired exams back as
training fuel.
Selected results (full ledger with methods, nulls, and receipts in
FINDINGS.md; artifacts in results/):
- Settled an open finite conjecture: degree-descending greedy coloring is NOT
optimal on connected triangle-free non-bipartite graphs (minimal
counterexample: 11 vertices, 13 edges — found by annealing, minimized, exact).
- Quantified the verifier-horizon regress: 99/121 conjectures with precision 1.0
at n<=6 are false at n=7/8; acceptance horizons must sit strictly inside
stress horizons or self-training loops train on artifacts.
- Measured salience vs relevance: the most salient memory is the counterfactually
useful one 4% of the time; a query-conditioned relevance score, 81%.
- Ran a full unattended continual-learning loop: cold-start 0/8 -> 8/8 in one
gated round; the multi-domain generation cracked a cross-domain frontier its
own examiner had exposed (base 1/33 -> gen-2 4/33, first MIS pass).
- Closed the same-bank scale comparison locally: stock Qwen2.5 1.5B->32B rose
8->20/48 while every stock model scored 0/24 on graph repairs; the task-routed
verifier-trained FastContext-4B scored 28/48 and earned PASS over its 22/48
base (6 gains, 0 regressions, exact p=.03125), repeated identically three times.
- Trained the first trajectory-disjoint Gen-4 engine student on 1,313 fresh
Stockfish/KataGo examples. Three fresh loads moved 7/69 -> 8/69, but the
item-level gate correctly BLOCKED promotion: 3 gains, 2 regressions, exact
p=1.0, including one historically stable regression. No promotion is claimed.
- The private promotion exam bank is deliberately NOT in this repo: publishing it
would let future models train on it — the leakage rule at internet scale.
Licensed AGPL-3.0. Commercial licensing available from the author.
Install the dependency-light core (CLI and stateless toolbox) with
pip install .. Optional integrations are grouped as agents, engines, and
local-models; pip install ".[all]" installs every integration.
The public HTTP MCP endpoint serves both the sessionless 2026-07-28
protocol (server/discover plus per-request metadata and headers) and the
initialize-based 2025-06-18, 2025-03-26, and 2024-11-05 revisions. The
stdlib server remains dependency-free; mcp>=2,<3 is needed only for SDK
clients and the stdio adapter.
Tagged releases contain archive-built wheels, source archives, SPDX 2.3 SBOMs,
and SHA256SUMS; GitHub Actions attaches build-provenance attestations to every
release artifact. For v0.8.0, verify the wheel against the repository with:
The product layer
The research harness above is operable as a product: one CLI, adapters for any
system under exam, pluggable verifiers, a CI contract, and agent-native
surfaces. Full guide in docs/product.md.
Grading an external endpoint burns every exposed item — permanently — via the
bank's exposure accounting; the promotion report carries paired evidence, an
exact McNemar p-value, the bank's own resolution statement, and a SHA-256
commitment to bank state. No surface, human or agent, ever serves exam item
contents.
The stateless toolbox turns eight mechanisms already in the repo into usable
file-in/receipt-out surfaces: Whetstone Inspector, Eval Leak Auditor, Promotion
Gate, Bank Health, SafePatch, Counterexample Hunter, Memory Relevance Debugger,
and Agent Replay Console. Each run exposes the decision path, an inspectable
visual certificate, a SHA-256 receipt, and a task-specific downloadable
artifact. Leakage analysis keeps exact identity, finite-corpus behavioral
equivalence, and text-similarity review as separate evidence tiers; only the
first two quarantine automatically. The public instance at
whetstone.cyberelf.link accepts only caller
supplied disposable data; it has no examiner bank, accounts, cookies, or upload
persistence. Private exams belong in the CLI or a customer-controlled instance.
The same public service now includes
Open Promotion Bench, a paired
scope-integrity track. open_bench_start gives a baseline and candidate the
same six procedurally varied virtual-repository tasks; open_bench_submit
checks requested state plus edit-scope conservation and counts gains,
regressions, and ties before issuing PASS/HOLD/BLOCK. A submitter may explicitly
publish a self-attested sanitized receipt. The ledger stores manifests,
transitions, counts, and hashes only—never task contents or answer patches—so
the existing workbench/report-card non-retention boundary remains intact.
Hosted report-card and Open Bench receipts are challenge-bound and signed with
an Ed25519 SSHSIG key. The one-command runners above fetch the HTTPS key bundle,
verify the issuer, challenge, content commitment, signature, and expiry, then
write the receipt. A valid signature authenticates Whetstone's grading result
and deployed build; it does not authenticate a caller-supplied model or harness
name. Runners fail closed on unsigned receipts unless --allow-unsigned is
explicitly used for a local development service. Busy report-card submissions
are retryable without consuming the session.
Support-panel minting currently fails closed by default: the lexical checker
passed its clean smoke corpus but failed 13/13 adversarial cases, so it cannot
mint promotion-capable support items until an independently reviewed hard
calibration is supplied explicitly.
The sixty-second demo
The whole promotion gate, live, on CPU, in about ten seconds:
It mints exam items at the verifier frontier, quarantines the ones a toy
student trained on (row-identity leakage check against a real buffer on disk),
grades two systems through the checker specs — a memorizer that echoes its
training data and a candidate that proposes repairs — computes per-item
discrimination, issues a gains-vs-regressions promotion decision, retires
saturated items into the downward-only training flow, and writes an append-only
ledger. The two systems are stored answer policies (this keeps the demo off the
GPU); every mint, quarantine, and grade is the production code path, computed
live. A canonical transcript is in
results/demo_investor_transcript.txt.
Research context
For readers (human or otherwise) mapping this to the literature: whetstone is an
experiment in continual learning and self-improvement for small language
models (QLoRA / parameter-efficient fine-tuning on consumer hardware), built to
resist the failure modes that usually kill such loops. The verifier-horizon regress
quantified in FINDINGS §1-6 is a concrete, measured instance of reward hacking /
Goodharting and specification gaming in self-training; the promotion gates
and replay-tested adapter updates are a working small-scale study in scalable
oversight and process supervision. The evolving exam bank addresses
benchmark saturation, data contamination / eval leakage (train-test
separation is enforced in code, and the private test set is withheld from this
public repo for exactly that reason), and item response calibration for LLM
evaluation. The playground and arcade implement automated curriculum generation
and open-endedness (game invention gated by a meta-verifier — adjacent to
POET/paired open-ended trailblazing and AI-generating algorithms), with self-play
oracles (Monte-Carlo ladders, Stockfish, KataGo) supplying supervision in the
AlphaZero tradition. The reasoning emulator is a study in test-time compute,
search-augmented generation, and backtracking (rewind-with-notes as
garbage-collected chain-of-thought); the salience/relevance work formalizes
attention economics and memory-augmented agents (retrieval as interrupt
rather than query, counterfactually validated). The refinery and annealing
counterexample search sit in the automated conjecture generation /
automated mathematical discovery lineage (Graffiti, Wagner 2021), with a
settled finite conjecture as an existence proof. Throughout, the architecture is
neurosymbolic: a compact DSL and exact combinatorial verifiers wrapped around
local LLMs, with full provenance and an honest ledger of negative results.
This repo grew from the blueprint in BLUEPRINT.md; the original scaffold notes follow.
What exists now:
- A JSONL-backed cognitive branch store.
- Typed commits/events with provenance.
- Branch creation, checkout, merge, diff, blame, and bisect.
- A patch-only Markdown editor with conservation hooks.
- A benchmark harness that injects long-document corruption and verifies that hooks catch it.
- A local Ollama-backed Markdown edit agent that writes only verifier-accepted patches.
Run the tests:
Run tests plus every current experiment:
Run a real local-model Markdown edit against the sample agreement:
Run the accept/retry/block usefulness benchmark:
Run the multi-document sequential edit benchmark:
Run Cognitive Git recall and coding benchmarks:
Export the seed Taste-RL preference and SFT datasets:
Generate a scaled synthetic Taste-RL dataset:
Run the first verifier-backed graph discovery loop:
Evaluate parseable graph conjectures from a proposal file:
Ask the local Ollama/LM Studio model to propose graph conjectures, then verify them:
Run a second model proposal pass conditioned on the previous verifier failures:
Run the larger research foundry comparison: stateless proposal loops versus an internal-git feedback loop that accumulates accepted conjectures, counterexamples, repairs, and SFT data:
Run the same foundry through the local Ollama/LM Studio model:
Run the foundry with a post-run scale check that stress-tests every accepted rule and
repair at larger n and commits scale falsifications to the ledger:
Run the self-contained frontier loop: FastContext 4B proposes on the GPU, the CPU
verifier kills, per-round scale falsifications feed back into the next proposal
prompt, and every proposal is judged for semantic novelty against the miner's
two-atom hull:
Hunt counterexamples by simulated annealing inside a conjecture's predicate class
(finds persist to the adversary library, which any pool can load with --library):
Run the model-vs-annealer counterexample duel:
Run the reasoning-emulator benchmark (save/load/rewind-with-notes/verifier-check as
model-invocable controls, vs linear CoT, verifier-scored):
Serve the Pro Action Replay MCP server (external agents can step, dump, and poke a
live reasoning session; registered in .mcp.json as reasoning-emulator):
Run the full conjecture refinery for a domain (enumerate -> verify -> stress ->
anneal-attack -> closure-expand library -> THEOREMS ledger + falsification museum):
Train a FastContext QLoRA smoke adapter from the foundry's accumulated verifier-repair data:
Generate a larger verifier-repair dataset for graph conjecture repair:
Train a FastContext QLoRA graph-repair adapter on JSON-only verifier targets:
Generate the hard repair dataset (the prompt contains counterexample evidence but not
the answer constraint, and heldout groups share no original expression with train):
Train and strictly evaluate the hard-task adapter (eval requires the output to be a
verifier-accepted strict refinement of the prompt's original expression, and reports
support retention plus mode-collapse diagnostics):
Stress-test every rule and repair accepted at n<=6 on larger graphs (random G(n,p)
plus deterministic adversaries: interleaved crown graphs and greedy-adversarial trees):
Build the stress-mined hard dataset, whose repair targets must also survive sampled
and adversarial graphs at larger n (targets stop being horizon artifacts by construction):
Run the first benchmark:
Run every current probe and write ledgers/training candidates:
Run only the local-model document probe:
Train the current learned document-corruption tripwire:
Export verifier-ledger training datasets:
Run the cached FastContext 4B QLoRA smoke trainer:
The benchmark is deliberately deterministic. It does not prove an LLM will edit safely. It proves the state and verifier layer can distinguish a clean patch from a corrupt full-document rewrite before a model is plugged in.
