Fittest

Fittest

A swarm of machine-learned interatomic potentials (MLIPs) that decides which model to trust on a structure, where, and when to stop trusting any of them and call DFT.

Survival of the fittest, in materials science. The models compete for the right to answer, and the fittest answer is served. Whether breeding new models on top pays off was tested against a pre-registered rule; it did not pass (results/evolution/SUMMARY.md).

Built at ScienceClaw 2026 (MIT Media Lab) for the NVIDIA ALCHEMI challenge, on NVIDIA ALCHEMI Toolkit 0.3.0-rc. Formerly TrustSwarm; the old GitHub URL redirects here.

• Code: this repo (fittest/ package, MIT licence)
• Hosted MCP server: https://fittest-mcp.westus2.cloudapp.azure.com/mcp
• Authoritative numbers: results/RESULTS.md (generated from the result files)

TL;DR

• Problem. Foundation MLIPs fail silently off-distribution. On held-out Transition1x reactions every model's force error rises toward the transition state; AIMNet2's median error at the TS is 2.1× its error at the reactant.
• Signal. Disagreement between independent foundation models (MACE-MPA-0, AIMNet2, Orb-v3) tracks error without labels (Spearman 0.73, 3,157 frames). Seed ensembles of one model do not measure error: their spread is 2.4–36× smaller than the error of their mean.
• Result (leak-free split, 200 DFT label lookups per arm). The swarm serves better answers: force RMSE 0.206 eV/Å vs 0.286–0.317 for every baseline, with a calibrated risk score (ECE 0.080 for arm D's P(served wrong); the hosted server's trust_score is a different quantity, see §1.1). The gain comes from discovering that averaging AIMNet2 and Orb is the best answer; a fixed AIMNet2+Orb average chosen in hindsight is as good (0.202).
• Negative result. The swarm ranks its own errors worse than the simple baselines: AUROC 0.73 vs 0.80–0.87. Its self-repair (LoRA fine-tune of MACE) halved force error on held-out labels but made relative energies worse, so its own gate refused to ship it.
• Evolution (pre-registered, negative on the primary metric). Letting the swarm breed, gate and prune its own fine-tuned models did not reliably beat the static swarm on served force error at the same DFT budget: the rule (paired CI, every seed, 3 breeding generations) fails in all three runs (A, B, C); it holds in 3 of 10 single seed runs. Reaction-relative energy error (secondary for run C) is lower than static in 9 of 10, but fine-tuning without selection gains as much, so that is not evidence for selection. A post-hoc fix (never prune base models, softer gate) was then pre-registered and tested on new seeds (run D): it held in 4 of 6 new seeds on CPU (the primary test; 5 required), so it is not confirmed; on GPU (three families, secondary) it held in 4 of 6 and was significantly worse than static in 2. results/evolution/SUMMARY.md
• Use it now. Add the hosted MCP server to Claude Code: claude mcp add --transport http fittest https://fittest-mcp.westus2.cloudapp.azure.com/mcp, then ask "Use fittest to check whether I can trust an ML potential on this water molecule: O 0 0 0, H 0.96 0 0, H -0.24 0.93 0." Structures are never stored (logging policy).

Contents

1. Use it now — MCP server, local install, Python API, ALCHEMI hook
2. How it works — problem, signal, agents, blackboard, evolution, ALCHEMI usage
3. Validation — data, split, metrics, ablation, figures
4. What failed and limitations
5. Reproduce
6. Repo map, privacy, licence, acknowledgements

1. Use it now

1.1 Hosted MCP server (no install)

fittest-mcp exposes the swarm to any MCP client over streamable HTTP. The three model agents (MACE-MPA-0, AIMNet2, Orb-v3) run as warm CPU processes on an Azure VM, and their disagreement is turned into a verdict with the Calibrator state learned in the final run (fittest/snapshots/trust_snapshot.json, identical to results/trust_snapshot.json). Landing page: https://fittest-mcp.westus2.cloudapp.azure.com/, public counts at /stats, health at /health. Full guide: docs/MCP.md.

Limits: ≤ 20 structures per call, ≤ 100 atoms each, non-periodic, coordinates within ±1000 Å, no atoms closer than 0.3 Å; element symbols must be exact (Cl or cl; PDB-style C1 is fine, ambiguous CA or HE1 is an error); per-client rate limit (burst 20, then 1 call per 2 s) plus a per-IP limit; 120 s per request; model evaluations are serialized.

The verdict rule. One rule feeds check_structures, explain and the usage counts. A structure gets trust only if all four hold:

1. every model agent supports its elements (AIMNet2 sets the limit; the full committee covers H B C N O F Si P S Cl As Se Br I), otherwise send_to_dft with reason unsupported_elements and no trust score;
2. the forces do not all vanish (max |F| < 0.05 eV/Å for every model means symmetry or exact equilibrium, where disagreement carries no information);
3. σ_F < τ;
4. trust_score ≥ 0.5.

Otherwise the verdict is send_to_dft, with every failing condition listed in verdict_reasons. τ = 0.620 eV/Å is fixed: it is the 70th percentile of σ_F on the round-0 pool, which the Calibrator's threshold rule never moved. trust_score is the Calibrator's Beta posterior mean of P(served answer correct) for the served candidate in the structure's (σ_F, mean |F|) bucket, built from DFT oracle labels on Transition1x with prior Beta(2, 1), so it is never exactly 1. It comes with trust_score_90ci (the 90% credible interval) and labels_behind_score (the DFT labels in that bucket); a bucket with few labels gives a wide interval, so read the score with it. "Correct" means force RMSE ≤ 0.25 eV/Å and relative energy error ≤ 0.25 eV (relative to the reaction's reactant). The ECE 0.080 in §3.3 is for arm D's risk score (max of the isotonic P(served wrong) and the Skeptic's p_S); the calibration of this per-bucket trust_score has not been measured beyond the 12-frame spot check below.

Claude Code

claude mcp add --transport http fittest https://fittest-mcp.westus2.cloudapp.azure.com/mcp

Cursor (~/.cursor/mcp.json)

{ "mcpServers": { "fittest": { "url": "https://fittest-mcp.westus2.cloudapp.azure.com/mcp" } } }

Claude Desktop / claude.ai: Settings → Connectors → Add custom connector → https://fittest-mcp.westus2.cloudapp.azure.com/mcp.

Example prompt: "Use fittest to check whether I can trust an ML potential on this water molecule: O 0 0 0, H 0.96 0 0, H -0.24 0.93 0." Atoms may be separated by newlines, commas or semicolons. In the Round-2 verification run this prompt, used verbatim, gave trust, trust score 0.857 (90% CI 0.61–0.99, 4 labels behind it), recommended AIMNet2, σ_F = 0.378 eV/Å below τ = 0.620 eV/Å (docs/MCP.md, verification log).

Hosted CPU vs the GPU run. On the 12 held-out Transition1x frames of results/service_demo.xyz the hosted server's σ_F ≥ τ flags agree 12/12 with the earlier GPU run (results/service_demo.json), with max |Δσ_F| = 0.0005 eV/Å. Under the verdict rule 7 of the 12 are trusted and none of the trusted frames is DFT-wrong; the earlier τ-only rule trusted 2 DFT-wrong frames.

Domain. The Calibrator learned from gas-phase C/H/N/O Transition1x reactions with at most 7 heavy atoms, including transition states and NEB paths. A domain_warning is added for elements outside C/H/N/O that the committee still supports, larger molecules, isolated atoms or fragments and unusually short bonds; there the trust score is an extrapolation. Absolute energies are on each model's own reference (PBE for MACE and Orb, ωB97M for AIMNet2), so served_energy_eV values from different models are not comparable and a batch whose served model differs carries an energy_warning. Compare reference_energy_eV (always AIMNet2) across structures instead.

1.2 Local MCP server

# demo only: a plain uvx install has no model agents
claude mcp add fittest-demo -- uvx --from git+https://github.com/Pavel-Tk/fittest fittest-mcp --fake-models

# real models: install into an environment that has them (documented, not yet tested end to end)
uv venv fittest-env --python 3.12
uv pip install --python fittest-env/bin/python "fittest[models] @ git+https://github.com/Pavel-Tk/fittest" \
    --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match   # CPU torch
# optional: Orb-v3 in its own env (orb-models conflicts with the Toolkit), as in scripts/setup_cpu.sh
uv venv orb-env --python 3.12
uv pip install --python orb-env/bin/python torch --index-url https://download.pytorch.org/whl/cpu
uv pip install --python orb-env/bin/python "dm-tree>=0.1.9" orb-models ase numpy \
    --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
# then register ONE of these two (same name, alternatives):
claude mcp add fittest-local -- $PWD/fittest-env/bin/fittest-mcp          # (a) MACE + AIMNet2 only
claude mcp add fittest-local -e FITTEST_ORB_PYTHON=$PWD/orb-env/bin/python -- $PWD/fittest-env/bin/fittest-mcp   # (b) full committee

A plain uvx --from git+… install has no model agents. It serves only --fake-models, CPU stand-ins whose verdicts mean nothing (useful to test the wiring); without that flag the tools return an error pointing to --fake-models, the [models] extra and the hosted server. For real verdicts, install into an environment that has the models and point your client at that environment's fittest-mcp. This real-models local install is documented but has not been tested end to end (fresh environment through to a verdict); the hosted server is the tested path.

The [models] extra installs MACE-MPA-0 and AIMNet2 through the ALCHEMI Toolkit. Orb-v3 needs its own Python environment because orb-models pins dependencies that conflict with the Toolkit; set FITTEST_ORB_PYTHON to that environment's Python. With only MACE and AIMNet2 the server serves a partial committee and every response carries a warning: τ and the Calibrator's buckets and ledger were built on the full three-model committee, so verdicts are approximate. Local mode runs over stdio on CPU: the commands above install CPU-only torch. --device cuda needs CUDA builds of torch in both environments instead (install without the CPU index, as scripts/setup_gpu.sh does). It spawns one process per installed model agent, each with its own random socket key, logs only to ~/.fittest-mcp/usage.sqlite and never reports anywhere.

1.3 Python API: the trust service

git clone https://github.com/Pavel-Tk/fittest && cd fittest
uv venv .venv --python 3.11 && source .venv/bin/activate   # Windows: .venv\Scripts\activate
uv pip install -e ".[dev,service]"

from fittest.service import TrustSnapshot, score, fake_predictors, connect_models

snap = TrustSnapshot.load("results/trust_snapshot.json")      # Calibrator state from the final run
# real model agents (one process each; GPU or CPU):
#   clients = connect_models(snap.models, base_port=6300, orb_python="/path/to/orb/env/python", device="cuda")
#   predictors = {m: c.predict for m, c in clients.items()}
predictors = fake_predictors(snap.models)                      # CPU stand-ins: wiring only, verdicts meaningless

water = {"Z": [8, 1, 1], "R": [[0, 0, 0], [0.96, 0, 0], [-0.24, 0.93, 0]]}
for r in score([water], predictors, snap):
    print(r["sigma_F"], r["flagged"], r["trust"], r["recommended_model"], r["action"])
# keys: sigma_F, sigma_F_max, mean_abs_fbar, flagged, trust, recommended_model, bucket,
#       action ("send to oracle / DFT" | "use <candidate>"), served_forces, served_members, per_model

HTTP and CLI wrappers of the same function:

python -m fittest.service serve --snapshot results/trust_snapshot.json --port 8000 [--fake-models]
curl -X POST localhost:8000/trust -H 'content-type: application/json' \
     -d '{"systems":[{"Z":[8,1,1],"R":[[0,0,0],[0.96,0,0],[-0.24,0.93,0]]}]}'
python -m fittest.service score my_molecules.xyz --snapshot results/trust_snapshot.json

score raises if a model agent the snapshot was learned with is missing: σ_F and its bins only mean something for the committee that produced them.

1.4 The ALCHEMI hook: DisagreementHook

A drop-in Toolkit hook for any FusedStage, FIRE2 or NEB run, with any committee of two or more models. At admission and every every steps it asks every committee member for forces (one round trip per model, concurrently), computes σ_F per system, and flags systems with σ_F ≥ τ. With graduate=True a flagged system gets status = exit_status, leaves the batch at the next refill_check, and the SizeAwareSampler refills its slot; the flagged system goes to your DFT queue.

from nvalchemi.dynamics import FIRE2, ConvergenceHook, FusedStage, SizeAwareSampler
from nvalchemi.dynamics.base import DynamicsStage
from fittest import paths
from fittest.hook import DisagreementHook
from fittest.model_server import spawn
from fittest.models.mace_batched import BatchedMACE

committee = {m: spawn(m, 6100 + k, python=paths.model_python(m)).connect()   # one process per model agent
             for k, m in enumerate(["mace_mpa0", "aimnet2", "orb"])}
flagged = []
hook = DisagreementHook(committee, tau=0.57,          # eV/Å; float, or a zero-arg callable read at every check
                        every=10, stage="both",       # pristine geometry at admission + every 10 steps
                        graduate=True,                # flagged -> leaves the batch, sampler refills the slot
                        id_map=lambda sid: ds.load_order[sid],
                        on_flag=lambda rec: flagged.append(rec["key"]))

driver = BatchedMACE().model                          # any Toolkit BaseModelMixin drives the dynamics
relax = FIRE2(model=driver, dt=0.05, n_steps=50, convergence_hook=ConvergenceHook.from_fmax(0.05))
for h in driver.make_neighbor_hooks():
    relax.register_hook(h, stage=DynamicsStage.BEFORE_COMPUTE)
sampler = SizeAwareSampler(ds, max_atoms=1024, max_edges=None, max_batch_size=48)
fused = FusedStage(sub_stages=[(0, relax)], sampler=sampler, sinks=[], refill_frequency=1, hooks=[hook])
with fused:
    fused.run(batch=None, n_steps=20000)

hook.records[flagged[0]]   # sigma_admission, checks[{step, sigma, sigma_max, tau, energy}], at_flag{Z, R, forces}
hook.summary()             # n_flagged, round trips, gather time, dead models, failures

ds is any dataset that follows the Toolkit sampler protocol; a complete example is in docs/REPRODUCE.md §2. For NEB use NEB(..., extra_hooks=[DisagreementHook(..., graduate=False)]); without dynamics use fittest.hook.DisagreementMonitor(committee, tau).check(keys, Z, R, batch, step=0, phase="x"). Committee members can be ModelClients, objects with .predict(Z, R, batch[, charge]), or plain callables returning (E, F). A member that raises or times out is logged and retired after max_failures consecutive failures; checks continue while two members answer. To add your own model as an agent, see docs/REPRODUCE.md §3.

2. How it works

2.1 The problem: silent failures where reactions are decided

Labs spend months deciding whether a foundation MLIP is good enough for their chemistry. The models return a number for every structure, with no warning when the structure is outside what they learned. On the 3,157 held-out Transition1x test frames (287 reactions), median force RMSE against DFT (ωB97x/6-31G(d)) rises from reactant to transition state for every model (results/fig/numbers.json, fig1):

(eV/Å, median per-frame force RMSE.)

[Force error rises toward the transition state]

2.2 The signal: disagreement between independent models

For M models predicting forces F_m on a structure with n atoms:

F̄_i  = mean_m F_m,i
s_i  = sqrt( mean_m ||F_m,i − F̄_i||² )      per-atom spread
σ_F  = mean_i s_i                            per-structure disagreement (eV/Å)

(fittest.signals.force_spread; the hook uses an identical vectorised twin, tested to agree.)

• Committee of independent models (MACE-MPA-0, AIMNet2, Orb): Spearman(σ_F, AIMNet2 force error) 0.73; the spread is about 1.8× the error.
• Seed ensemble (the four shipped AIMNet2 members): Spearman 0.68, but the spread is about 0.4× the error — overconfident.
• On the 1,485 eval frames: the AIMNet2 ensemble's median error is 2.4× its spread; four MACE LoRA seeds (arm B′) are 36× under-dispersed (spread 0.008 eV/Å, error 0.278 eV/Å).

Seed ensembles can rank errors well while being badly under-dispersed: ranking and calibration are different properties. This is why the swarm uses models trained on different data by different groups.

[Spread vs error]

2.3 The agents and the blackboard

Each agent holds private information and talks to the others only through an append-only blackboard (fittest.blackboard.Blackboard; the full arm-D log is results/blackboard_armD.jsonl, 5,921 events). The round driver advances the clock and makes no decisions; there is no central planner.

flowchart LR
    S["Stream<br/>label-free structures"] --> P["ModelAgents x3<br/>MACE · AIMNet2 · Orb<br/>(one process each)"]
    P -->|"prediction"| M{"Monitor / DisagreementHook<br/>σ_F ≥ τ ?"}
    M -->|"pass"| C["Calibrator<br/>ledger · verdict · trust"]
    M -->|"flag"| B["Bids<br/>ModelAgents + Skeptic"]
    K["Skeptic<br/>private references,<br/>planted failures"] -->|"veto, audits"| B
    C -->|"verdict"| K
    B --> O["Oracle<br/>k DFT labels / round"]
    O -->|"oracle_result: true error<br/>of every candidate"| C
    C -->|"reputation"| P
    C -->|"threshold τ"| M
    C -->|"worst model + labels"| U["Curator<br/>LoRA fine-tune + gate"]
    U -->|"finetune_report"| C

What the Calibrator serves. Candidates are every non-empty subset mean of the models (3 singletons, 3 pairs, the 3-way consensus). The context bucket is label-free: σ_F quantile bin (4) × mean |F̄| bin (3). Per bucket it serves the candidate with the highest posterior probability of being correct. Its risk score is an isotonic P(served answer wrong | feature); the feature (σ_F or the spread among the served candidate's members) is chosen by 5-fold reaction-grouped CV on its own labels (it chose σ_F: CV AUROC 0.756 vs 0.752). τ = smallest σ with ledger P(wrong | σ ≥ τ) ≥ 0.5, floored at the round-0 70th percentile of σ_F. In the final run τ stayed at that floor (0.620; see §4).

What happened in the final run (arm D). 200 oracle calls: 50 exploration audits and 150 won by bids (9,745 bids placed). The hook flagged 601 systems; the Skeptic vetoed 1,788 trusted verdicts (1,202 on pool systems; vetoes on eval are advisory). On eval the Calibrator served mean(aimnet2+orb) on 1,373 of 1,485 frames and aimnet2 on 112. Final reputations were nearly equal (aimnet2 0.89, mace_mpa0 0.89, orb 0.89, skeptic 0.86).

2.4 The evolutionary layer

Status: done; the pre-registered claim does not hold. Three runs (A: MACE LoRA offspring; B: AIMNet2 offspring; C: niche specialists of all three families at budgets 200/400/800), two seeds each. Full write-up, controls and exploratory follow-ups: results/evolution/SUMMARY.md.

The static swarm above has a fixed population. The evolutionary loop (fittest/evolution.py, scripts/evolve.py) asks whether a population that breeds, selects and prunes its own members improves at the same DFT budget as a static control. All settings below were fixed before the real run (results/evolution/METHODS.md). One generation:

flowchart LR
    T["1 Triage<br/>population predicts the pool;<br/>Calibrator chooses what to serve"] --> O["2 Oracle<br/>50 labels per generation:<br/>75% highest σ_F, 25% audits of trusted frames"]
    O --> F["3 Fitness<br/>marginal contribution<br/>answerer Δ_ans · critic Δ_crit"]
    F --> S["4 Selection<br/>prune members whose<br/>contribution CI includes 0"]
    S --> B["5 Breeding<br/>Curator mutates the fittest<br/>MACE recipe; clones rejected"]
    B --> G["6 Gate<br/>child vs parent on select labels<br/>+ ALCHEMI acceptance report"]
    G --> H["7 Hot-swap<br/>survivors + gated offspring;<br/>ledger re-fitted"]
    H --> T

• Budget and controls. 4 generations × 50 Transition1x lookups = 200, the same budget as arm D. The static control runs the identical triage/oracle procedure with the population fixed at MACE-MPA-0 + AIMNet2 + Orb. Every generation of every run is scored on the same eval frames with 1,000 shared reaction-bootstrap resamples, so comparisons are paired.
• Breed / select split. The pool's 16 endpoint-geometry components are split once into select (7 components, 572 frames, 52 reactions) and breed (9 components, 1,100 frames, 100 reactions); each generation labels 20 select + 30 breed frames. Offspring train only on breed labels; fitness and the gate use labels no member trained on, so selection is never measured on a child's own training geometry.
• Fitness = marginal contribution to the swarm, on clean labels:
  • answerer: Δ_ans(m) = err(P \ m) − err(P), where err is the mean served error in wrongness units (F_RMSE / 0.25 + |ΔE_rel| / 0.25) and the served answer comes from a Calibrator fitted out-of-fold (5 reaction-grouped folds), so a member cannot look useful by overfitting the labels it is scored on;
  • critic: Δ_crit(m) = AUROC(σ_F(P) → P wrong) − AUROC(σ_F(P \ m) → P\m wrong);
  • 400 paired reaction-bootstrap resamples give each Δ an SE and 95% CI; fitness = Δ_ans/SE_ans + Δ_crit/SE_crit; the role (answerer or critic) is the larger z-score.
• Selection. Recomputed after every death. A member is eligible when neither Δ has a CI lower bound above 0 and fitness ≤ 0.5; the least fit eligible member dies while the population exceeds 3, then the least fit dies while it exceeds 6.
• Breeding. 3 children per generation (generations 0–2) from the fittest MACE-family member(s); the foundation MACE-MPA-0 is the germline if none is alive. Mutations from the parent recipe (E:F loss weight 10:10, LoRA rank 16, α 2, lr 10⁻³, 40 epochs): child a energy weight ×3; child b energy weight ×10 and LoRA rank ×½ or ×2; child c learning rate ×½ or ×2, epochs −10/+20, 80% reaction subsample. A child of an offspring starts from its parent's merged weights. Clone rejection: Pearson r > 0.95 between the child's and a member's deviations from the population-mean force, or median per-frame force RMS difference < 0.02 eV/Å.
• Gate (select labels, component-disjoint from the child's training labels, asserted twice): force RMSE ≤ parent; reactant-relative ΔE MAE ≤ parent; ALCHEMI build_acceptance_report from-scratch ratio (child vs parent) ≤ 1.01. The 1% tolerance was set after a smoke test and before the real run: the Toolkit's energy-per-atom MAE is dominated by frozen E0 offsets identical for child and parent (ratio 1.000008), so at exactly 1.00 the check is a float-precision coin flip. The Toolkit's own verdict at 1.0 is recorded too. Failing children are stillborn and kept in the family tree.
• Differences from arm D, stated so the two are not confused: no Skeptic kNN vetoes, no bid auction (σ-ranked oracle + audits instead), no planted cases; the AUROC risk score is the population's σ_F.
• Eval never feeds back: only the final scorer (EvalScorer) reads eval labels.
• Decision rule (fixed in advance, scripts/write_evolution_summary.py): the flagship claim holds only if, in every seed run, served force RMSE at the final generation is significantly lower than both the run's own generation 0 and the static swarm at the same budget, over ≥ 3 generations, with offspring in the final population and no significant regression vs static in relative-energy MAE or ECE.

Results (final generation minus static at the same budget, served on eval, paired 95% CI; * = CI excludes 0). Run C used the same engine with niche specialists, a niche gate and no ALCHEMI acceptance check (results/evolution_c/PREREG.md).

Verdict: the flagship claim does not hold under the pre-registered rule (it needs every seed). Most children fail the gate on relative energy (run A: 15 of 18 stillborn; GPU run C: 36 of 54 stillborn, 35 of them on rel-E). Why the evolved swarm sometimes ends up worse than static is not established: base models were pruned in the worse runs but also in most of the runs that passed. The paired CIs cover eval resampling only; run-to-run noise is about 0.02 eV/Å (the static swarm alone moves 0.202 → 0.220 between seeds). Fine-tuning without selection looked more consistent in exploratory runs, but it changes several things at once and lost to the evolved arm in one of the two pre-registered seeds. A post-hoc fix (protect the base models, tolerate 10% worse rel-E at the gate) was pre-registered as run D (results/evolution_d/PREREG.md) and run on new seeds 6–11: on CPU (primary, AIMNet2 specialists) it holds in 4 of 6, below the pre-registered 5 of 6, so it is not confirmed; on GPU (secondary, three families) it holds in 4 of 6 and two seeds are worse than static. Family trees: results/fig/fig8_family_tree_run{A,B}_s{0,1}.png; per-generation curves: results/fig/fig7_evolution_run{A,B}_s{0,1}.png; budget axis: results/evolution_c/fig9_budget.png.

2.5 How the ALCHEMI Toolkit is used

In-flight screen (600 pool frames, MACE-MPA-0 driver, τ = 0.571 = 70th percentile of cached σ_F, check every 10 steps, ≤ 50 FIRE2 steps per system, L4 GPU):

Batched NEB (8 reactions, AIMNet2 driver, climbing image, all 8 converged in 21 s): along the DFT reference bands, σ_F follows force error (median per-reaction Spearman 0.96, pooled 0.85; two reactions at 0.12 and 0.16). σ_F does not track relative-energy error (median Spearman 0.14, pooled −0.20). The maximum of σ_F is at the DFT transition state in 3/8 reactions and within one image of it in 6/8.

[NEB]

3. Validation

3.1 Data and split

• Transition1x (ωB97x/6-31G(d); C, H, N, O; neutral; ≤ 7 heavy atoms). 5,632 frames: reactant, product, TS and sampled NEB path images.
• The official Transition1x split is by chemical formula. Inside the test formulas, Transition1x reuses reactant geometries across reactions, so a per-reaction split leaks: it had 732 near-duplicate endpoint geometries between pool and eval. The final split groups reactions into connected components that share any reactant or product geometry (37 components, 287 reactions) and splits by component: 0 shared endpoint geometries and 0 near-duplicates (max |ΔR| < 10⁻³ Å) between pool and eval (results/split_report.json).

Pool and eval still share chemical formulas.

• Models and leakage. MACE-MPA-0 (MPtrj + sAlex, PBE/PBE+U), AIMNet2 (ωB97M-D3, member 0 of the shipped ensemble), Orb orb-v3-conservative-inf-omat (OMat24, PBE/PBE+U). Orb models trained on OMol25 were excluded: OMol25 recomputed Transition1x geometries. Details: docs/data_and_models.md.

3.2 Metrics, fixed before running

• Wrong ⇔ force RMSE > 0.25 eV/Å or |ΔE_rel| > 0.25 eV (relative to the reaction's reactant, so per-model reference energies cancel). Sensitivity at 0.15 and 0.4 is in ablation.json (by_threshold).
• AUROC: does the arm's warning score rank its own wrong served answers? Also against a common target (AIMNet2 wrong), since AIMNet2 is in every arm.
• Recall@K: share of wrong served answers that received an oracle call.
• Planted caught: share of planted known failures (AIMNet2 force RMSE > 0.4) flagged, vetoed or checked.
• Served error on eval (force RMSE, relative-energy MAE), ECE and Brier of the arm's P(wrong), silent failures (badly wrong, F-RMSE > 0.4, and trusted without a check), trusted share.
• 95% CIs: 1,000 bootstrap resamples over reactions, paired across arms.
• Budget: 200 Transition1x label lookups per arm (40 per round × 5 rounds).

3.3 Ablation (final run, v4)

Arm D is D_fittest in results/ablation.json. Result files written before the rename said D_trustswarm; only that label was changed, no number.

Full table with every CI and the pool metrics: results/RESULTS.md.

Fittest (D) vs each baseline, paired 95% CI (better / worse: the CI of the difference excludes 0):

What is significant

• Served answers: Fittest's force RMSE (0.206 eV/Å) is better than every baseline, and it is wrong on 40% of eval frames vs 54–73%. The gain is the discovery of mean(aimnet2+orb) from 200 labels; no baseline had subset candidates. A fixed AIMNet2+Orb average chosen in hindsight with eval labels scores 0.202 eV/Å, i.e. as good. Budget sweep: 0.202 / 0.206 / 0.204 eV/Å at k = 20 / 40 / 80 labels per round.
• Calibration: ECE 0.080 and Brier 0.185, better than A_single, A_random, B, B′ and C; tied with C_audit. This is for arm D's risk score; the hosted MCP server's trust_score (per-bucket Beta posterior) is a different quantity, whose calibration has not been measured beyond a 12-frame spot check (§1.1). The calibration gain comes from the Skeptic's random audits of trusted answers.

What is not

• Error ranking: Fittest's AUROC (0.730) is significantly worse than A_single, B, B′, C and C_audit.
• Silent failures: Fittest has 0/96 on eval and 1/115 on pool, not significantly different from the five non-random baselines — four of which trust nothing and have zero by construction.
• Planted failures: 0.45 caught vs 0.33–0.70; better than C_collective only.

[Ablation]

Why averaging AIMNet2 and Orb helps (eval, force-only wrongness at 0.25 eV/Å): AIMNet2 alone is wrong on 57.2% of frames, the AIMNet2+Orb mean on 26.3%; adding MACE undoes it (54.3%); the AIMNet2 seed mean reaches 51.6%.

[Best subset]

3.4 Self-repair (Curator)

The Curator chose mace_mpa0 (highest ledger force RMSE, 0.80 eV/Å vs 0.51 Orb and 0.43 AIMNet2) and handed over 186 oracle labels; the GPU job trained a LoRA adapter on 160 and held out 26 (3 whole geometry components, 0 shared geometries with training).

Not shipped. The unshipped adapter's median force RMSE on eval would have been 0.279 vs 0.558 eV/Å for the base model (figure only; never served).

[Repair]

4. What failed and limitations

Everything below is also in results/RESULTS.md ("Failures and caveats").

• Error ranking is worse. AUROC 0.730 vs 0.80–0.87 for the simple baselines. Arm D's risk score is learned from bid-won labels, which are concentrated at high σ.
• Selection bias in the labels. In v1 the oracle labelled only flagged (high-σ) systems, so trust was never tested where it was given. v2 added the Skeptic's random audits of trusted systems (25% of each round's budget); they reduce but do not remove the bias.
• Leaks, found and fixed. v1–v2 split by reaction; Transition1x reuses reactant geometries, so pool and eval shared 732 near-duplicate endpoints. v3 splits by geometry component (0 shared). The Curator's first hold-out was also leaky (9 hold-out reactions each shared an endpoint with training) and was replaced by whole components. v1's planted cases were defined as "MACE wrong", which favoured arms containing MACE; v2 onward uses "AIMNet2 badly wrong".
• Repair blocked by its own gate. Forces improved 1.355 → 0.692 eV/Å, relative energies got worse (0.314 → 0.364 eV), and the ALCHEMI from-scratch ratio was 1.404 > 1.0.
• τ never moved. The Calibrator's threshold stayed at its round-0 floor (0.620) for all rounds: flags came from the 70th-percentile rule, not from learning.
• Reputations barely differentiated (0.86–0.89): reputation-weighted bidding behaved like summed bids.
• The served-answer gain equals a hindsight choice (0.202 vs 0.206 eV/Å). Which candidate each budget in the sweep recommended is not recorded.
• Level of theory. MACE-MPA-0 and Orb-v3-omat are PBE materials models, AIMNet2 is ωB97M-D3, labels are ωB97x/6-31G(d). Absolute energies are never compared across models; part of every error is the level-of-theory gap, not model failure.
• Labels are lookups, not live DFT. The Oracle reveals stored Transition1x labels.
• The ablation is a CPU replay of GPU predictions. Agents replay predictions cached by the model-agent processes on the GPU, and the hook's logic runs as its CPU twin (DisagreementMonitor). The ALCHEMI hook itself runs in the 600-frame in-flight screen and in NEB; the ablation does not consume that screen's oracle queue.
• Narrow calibration domain. Trust numbers are learned on C/H/N/O reaction chemistry with ≤ 7 heavy atoms. On water, metals, charged or periodic systems they extrapolate.
• Evolution. The pre-registered rule needs a win in every seed; with two seeds per condition it has little power, and the wins we saw were seed-dependent. Most children fail the gate on reaction-relative energy. Run C's first GPU attempt is invalid (OOM children counted as stillborn) and is not quoted; the clean GPU rerun (three families) is the faithful run C, and C′ (CPU, AIMNet2 only) is a reported deviation. Extra seeds and design variants are exploratory, run on the same eval split, and labelled so. Details: results/evolution/SUMMARY.md.
• The synthetic smoke test (scripts/make_synthetic.py) shows the pipeline runs; it is not evidence.

5. Reproduce

5.1 CPU, from a fresh clone (minutes)

git clone https://github.com/Pavel-Tk/fittest && cd fittest
uv venv .venv --python 3.11 && source .venv/bin/activate
uv pip install -e ".[dev,service]"
pytest                                  # CPU suite; GPU-only hook tests skip
python scripts/make_synthetic.py        # synthetic frames + predictions
python scripts/run_ablation.py --frames data/synthetic/frames.pkl --preds data/synthetic/preds --out results/synthetic
python -m fittest.service serve --snapshot results/trust_snapshot.json --fake-models --port 8000

CPU VM with the real model agents (what the hosted MCP server runs: Ubuntu 24.04, 8 vCPU, no GPU):

bash scripts/setup_cpu.sh [WORKDIR]     # core env (CPU torch + ALCHEMI Toolkit 0.3.0-rc + MACE + AIMNet2) and an Orb env

The cached predictions (data/frames.pkl, data/preds/, data/ckpt/, 23 MB) are gitignored; copy them from a machine that ran step 1 below. Every CPU result step (2, 5, 7, figures, RESULTS.md) then runs without a GPU.

5.2 GPU (Colab L4), every result file

bash scripts/setup_gpu.sh ..            # toolkit 0.3.0-rc (cu13) + core env, Orb env, Transition1x (6.6 GB, md5-checked)

The arm-D label set is md5-pinned (ad5f14229a51679267d4fd9456feb589, docs/STATUS_core.md); the v3 and v4 reporting changes did not alter it. Environment variables (FITTEST_T1X, FITTEST_CORE_PYTHON, FITTEST_ORB_PYTHON, FITTEST_NEB_DIR, FITTEST_LOG_DIR) and per-step details: docs/REPRODUCE.md.

6. Repo map

fittest/
  agents/           ModelAgent, Oracle, Skeptic, Calibrator, Curator, PoolStream, DisagreementMonitor
  blackboard.py     append-only event log (JSONL audit trail)
  signals.py        σ_F, errors, wrongness
  candidates.py     subset-mean candidates the Calibrator can serve
  hook.py           DisagreementHook (ALCHEMI Toolkit) + toolkit-agnostic DisagreementMonitor
  model_server.py   one process per model agent (own Python env if needed) + client
  models/           Toolkit wrappers (batched MACE, AIMNet2 calculator, I/O)
  mace_lora.py      LoRA checkpoint format + loader
  ablation.py       arms, metrics, paired bootstrap
  round_driver.py   the clock
  service.py        trust service: score(), FastAPI app, CLI
  mcp_server.py     fittest-mcp (MCP tools: check_structures, explain, swarm_status, usage_stats)
  usage_log.py      privacy-preserving usage log of the MCP server
  evolution.py      evolutionary loop (fitness, selection, breeding, gate)
  snapshots/        packaged Calibrator snapshot used by the MCP server
scripts/            GPU and CPU job scripts (see §5), setup_gpu.sh, setup_cpu.sh
results/            every number in this README; RESULTS.md; fig/; earlier versions v1–v3
docs/               DESIGN.md (contract), REPRODUCE.md, alchemi_api_notes.md, data_and_models.md, MCP.md
tests/              CPU pytest suite (synthetic data; no GPU or Transition1x needed)
deploy/             hosted MCP: Caddy (TLS) + systemd units for fittest-mcp and warm CPU model agents

Privacy: what the hosted MCP server logs

Every tool description, the server instructions and every check_structures response summarise this policy (fittest/usage_log.py, docs/MCP.md):

• Logged per call (one SQLite row): UTC time, tool (or unknown), transport, an anonymous client id, an internal-test flag, structure count, atom count, verdict counts, per-structure failures, latency, ok/error and an error category (input_error, invalid_arguments, unknown_tool, not_ready, rate_limited, busy, timeout or the exception type; never the message, which can echo input). Every call is logged, including unknown tools and schema-validation failures. Hosted, the log is SQLite at /var/lib/fittest-mcp/usage.sqlite, owned by the unprivileged fittest-mcp system user.
• Client id = HMAC-SHA256(secret salt, client name | remote IP), truncated to 16 hex characters. The client version is deliberately not part of it, so upgrading or forging a version string does not create new ids. The salt is a random server-side file (/var/lib/fittest-mcp/salt, mode 600, never committed), so ids cannot be reversed. Hosted, the IP comes from Caddy, which overwrites X-Forwarded-For.
• Raw IPs are not stored by the app or written to the access log: uvicorn's access log is off and Caddy's access log drops the remote IP, client IP, X-Forwarded-For, X-Real-IP, Forwarded, Cf-Connecting-Ip and True-Client-Ip fields. Since 2026-10-31 19:11 UTC Caddy's error and unmapped-host logs drop them too, and requests for any other host name (e.g. the bare IP) get a 404 and are not logged. Before that, 140 Caddy entries in the system journal contained client IPs (9 distinct IPs: 2 error entries from 502s during restarts, 138 bare-IP requests, mostly scanners); they stay there until journal rotation.
• Never stored or logged: structures, coordinates, element lists, structure names, results. The server's own journal lines carry only the tool name and an error category or exception type. A test scans the raw database bytes for a planted structure name and coordinates; another checks that a failing call leaves no input in the log.
• Users = distinct client ids that received at least one structure verdict from check_structures or explain, excluding internal test clients (header X-Fittest-Internal: 1; a client id seen with it is registered as internal). Calls to swarm_status or usage_stats, and calls whose structures all failed to parse, do not count. Every test so far was marked internal. Caveats: people who share a public IP and client name merge into one user (an office or venue Wi-Fi, and claude.ai / Claude Desktop custom connectors, which call from Anthropic's shared egress IPs, so all claude.ai users may count as one); one person on several networks or clients counts several times; varying the client name from different IPs could still inflate the count. Read it as a rough activity signal, not a count of people. Aggregate counts are public at /stats.
• A privacy bug, found and fixed: the first deploy left uvicorn's access log on, which wrote client IPs into the systemd journal. It was turned off and the journal vacuumed; a re-check found the test client's IP 0 times in the server and Caddy journals and in the Caddy access log.
• Local (stdio) mode logs only on your machine (~/.fittest-mcp/usage.sqlite).

Licence

MIT (LICENSE). Model weights and datasets keep their own licences.

Acknowledgements and references

• NVIDIA ALCHEMI Toolkit 0.3.0-rc — in-flight batching, hooks, batched NEB, LoRA fine-tuning and acceptance reports. https://github.com/NVIDIA/nvalchemi-toolkit
• Transition1x — M. Schreiner, A. Bhowmik, T. Vegge, J. Busk, O. Winther, "Transition1x – a dataset for building generalizable reactive machine learning potentials", Sci. Data 9, 779 (2022). doi:10.1038/s41597-022-01870-w
• MACE / MACE-MP — I. Batatia et al., "A foundation model for atomistic materials chemistry", arXiv:2401.00096; I. Batatia, D. P. Kovács, G. N. C. Simm, C. Ortner, G. Csányi, "MACE: Higher order equivariant message passing neural networks for fast and accurate force fields", NeurIPS 2022. Checkpoint medium-mpa-0.
• AIMNet2 — D. M. Anstine, R. Zubatyuk, O. Isayev, "AIMNet2: a neural network potential to meet your neutral, charged, organic, and elemental-organic needs", Chem. Sci. (2025). doi:10.1039/D4SC08572H
• Orb-v3 — B. Rhodes et al., "Orb-v3: atomistic simulation at scale", arXiv:2504.06231. Checkpoint orb-v3-conservative-inf-omat.
• LoRA — E. J. Hu et al., "LoRA: Low-rank adaptation of large language models", arXiv:2106.09685.
• Closest prior art on heterogeneous committees of foundation MLIPs: Y. Liu et al., "Heterogeneous ensemble enables a universal uncertainty metric for atomistic foundation models", arXiv:2507.21297. Query-by-committee active learning: J. S. Smith et al., J. Chem. Phys. 148, 241733 (2018). More in docs/data_and_models.md.