Studydiff

StudyDiff

Understand why two scientific studies reach different conclusions —
with every claim verified against the source.

[CI] [License: MIT] [Node 20+] [Live demo] [Built with Claude] [MCP server] [Glama MCP server] [Listed on mcpservers.org] [npm] [npm downloads] [Built with Claude: Life Sciences hackathon] [PRs welcome]

Live demo · Quick start · How it works · Examples · MCP server · Roadmap · Contributing

[StudyDiff comparing two studies and explaining why they disagree]

Two well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. StudyDiff does that digging. Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.

It is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.

Why it's different

Most "AI literature" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:

• It shows the evidence, then gets out of the way. The design differences up top; every value with the verbatim sentence that supports it underneath — and click any verified value to see that exact sentence highlighted in the paper's own text.
• It refuses to guess. Any field the source doesn't state is shown as not reported, never inferred.
• It verifies itself. A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded before it can be used as a reason.
• It knows what it can't do, because it was measured — on two independent benchmark sets, the second one held out and scored once. See below.

Does it work? A measured answer

StudyDiff used to rank the divergent design dimensions and present the top one as the likely driver of a disagreement. We built a benchmark to test that, and it doesn't work.

15 documented contradictions where the literature has since established why the papers disagreed — each label carrying its own citation, the set built blind before any accuracy number existed. Scored against StudyDiff's top-ranked driver:

Top-1 accuracy (strict)      13.3%  (95% CI 3.7-37.9%)   [2/15]Baseline "always say assay"  13.3%  (95% CI 3.7-37.9%)   [2/15]                             → discordant on 0 of 15 casesOracle ceiling (reachable)   66.7%  (95% CI 41.7-84.8%)  [10/15]Non-assay-labelled cases      0.0%  (95% CI 0.0-22.8%)   [0/13]

The ranking was a fixed prior (DRIVER_RANK in src/compare.mjs) in which assay outranks everything. Two papers almost always use somewhat different methods, so assay almost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly the two assay-labelled cases. It is not merely as good as guessing a constant; it is behaviourally identical to it on every case in the set.

Fixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive rejections doubled the oracle ceiling from 33% to 67% — the established cause is now an available candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at all. The ranker was handed the right answer five more times and took none of them.

What changed as a result: the app no longer nominates a primary driver. It presents the divergent dimensions as an unranked list, because that list is informative (it contains the established cause 10 times in 15) while the ordering is not. Choosing among them needs domain knowledge the tool doesn't have.

What still holds: which dimensions differ, which are identical (ruled out), and the verbatim sentence behind every value. None of that depends on the ranking.

Confirmed blind, on a second set

Those 15 cases had by then been read across two phases — failures analysed, spans re-audited — so every post-fix figure from them is development-set accuracy, not a blind measurement. So we built a second set and measured it once.

eval/cases-heldout.json is 15 further documented contradictions, curated to a protocol written and committed before any case was selected (eval/HELDOUT-PROTOCOL.md), by a curator kept blind to the dev set's per-case failures, in deliberately different fields: microbiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease. No paper and no contradiction is shared with the dev set — selftest enforces that mechanically.

Top-1 accuracy (strict)      13.3%  (95% CI 3.7-37.9%)   [2/15]Baseline "always say assay"  20.0%  (95% CI 7.0-45.2%)   [3/15]                             → discordant on 1 of 15 casesOracle ceiling (reachable)   73.3%  (95% CI 48.0-89.1%)  [11/15]Non-assay-labelled cases      0.0%  (95% CI 0.0-24.3%)   [0/12]

On unseen data the prior scores below the constant guess — by exactly one case. The intervals overlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is that it remains indistinguishable from guessing assay every time, not that it is worse.

The line that does not move is the last one. Across both sets, 25 cases where the established cause was something other than assay, the prior identified none of them. And the ceiling here is higher than on the dev set — 73.3% against 66.7% — so extraction put the right answer in front of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion reproduced on data the development loop never saw, which is the only way it could have been strengthened.

The number is reported as-is, and is never pooled with the dev-set number: summing them into an "n=30" figure would relaunder read data as blind data. Nothing in src/ was changed on the basis of it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean pass — the first run reported n=14 after one paper failed to fetch, and the full set was scored after retrieving it. Both figures, the sixteen defects an adversarial verification pass found and corrected before any scoring, and the reasoning behind each label are recorded in the file's own provenance block rather than summarised away.

npm run eval:heldout            # the blind number, offline and freenpm run eval:selftest:heldout   # set integrity, incl. zero overlap with the dev set

Full method, the pre-registered decisions, and every prediction that turned out wrong: eval/README.md and eval/PHASE2.md. The benchmark sets are eval/cases.json (development) and eval/cases-heldout.json (held-out).

npm run eval            # offline, free, no API key — regenerates the numbers abovenpm run eval:selftest   # validates the harness maths and set integrity

eval/cache/ and eval/cache-heldout/ are committed on purpose. They aren't build output, they're evidence: the published numbers are reproducible from artefacts in the repo rather than taken on faith.

Quick start

Under 60 seconds, no API key, no network:

git clone https://github.com/nickjlamb/studydiff && cd studydiffnpm installnpm run demo                        # explains a real, famous contradictionnpm run demo -- resveratrol-sirt1   # a second worked examplenpm run demo -- treg-stability      # a third: Treg lineage stability

Run the web app:

cp .env.example .env    # add ANTHROPIC_API_KEY for live comparisonsnpm run serve           # http://localhost:4173

The built-in examples run on cached data and need no key. To compare live, add your key and use the PMID / DOI, Upload PDF, or Paste inputs.

How it works

flowchart LR  IN["Two papers<br/>PMID · DOI · PDF · text"] --> R["Retrieve<br/>PubMed / PMC / PDF"]  R --> E["Extract<br/>Claude → structured study cards"]  E --> V["Verify<br/>deterministic grounding"]  V -- "ungrounded → not reported" --> E  V --> C["Compare<br/>divergent vs. shared design"]  C --> X["Present<br/>divergent dimensions + ruled out"]

1. Retrieve — a PubMed/PMC client with a full-text-to-abstract fallback that tags how deep it read (fulltext / abstract / pasted); uploaded PDFs are text-extracted server-side.
2. Extract — Claude turns each paper into a fixed study card (species, model, assay, dose, timing, endpoint, sample size, statistic, finding, limitations). Every field carries a verbatim supporting quote; absent fields default to not reported.
3. Verify — grounding runs first: any value whose quote isn't in the source, or whose numbers don't trace, is downgraded to not reported. StudyDiff can't cite a fact it hasn't verified.
4. Compare — deterministic: which design dimensions are reported by both papers, which of those diverge, and which are identical. Divergence is a token-overlap test on design dimensions and strict inequality on the conclusion itself, because a conclusion's polarity matters where a method list's content does.
5. Present — the divergent dimensions unranked, each with both papers' values and the verbatim sentence behind them, and the identical ones explicitly ruled out. StudyDiff does not nominate one as the cause: it was measured and it doesn't work.

Two things hold the guarantee up: the API key never leaves the server, and verification is deterministic — grounding and comparison involve no model judgment at all, so the same verified evidence always yields the same divergent dimensions.

Why Claude?

StudyDiff's reliability comes from how it uses Claude, not just that it does:

• Structured extraction via tool-use. Each paper becomes a study card through a forced Claude (Sonnet) tool schema, so every field returns validated and carries a verbatim supporting quote — no free-text parsing, no "mostly-JSON" failures.
• Reasoning that maps claims to evidence. Claude reads the methods/abstract prose and identifies both the design value and the exact sentence that supports it — the hard part of turning unstructured papers into comparable, auditable cards.
• Claude proposes, grounding disposes. StudyDiff never uses an LLM as the judge. A deterministic check verifies Claude's output against the source and downgrades anything unsupported before it's shown. Pairing Claude's tool-use extraction with non-LLM verification is what lets the tool trust its own output and sidesteps the usual LLM-as-evaluator pitfalls.
• Built with Claude Code. The whole app — pipeline, UI, hardening, deploy — was built iteratively with Claude Code in a single agentic loop.

Examples

All three ship as offline demos (npm run demo / npm run demo -- resveratrol-sirt1 / npm run demo -- treg-stability).

Using it

• Web app (npm run serve) — examples, PMID/DOI lookup, PDF upload, or paste; streams each pipeline step live, lets you click any grounded value to see it highlighted in the source, and exports a reproducible Markdown report with every value's verbatim supporting sentence.
• MCP server (npm run mcp) — lets Claude, or any agent, call the contradiction engine directly. See MCP server.
• CLI — node src/cli.mjs --q "Does resveratrol activate SIRT1?" 12939617 19843076
• Deploy — see DEPLOY.md (Railway + custom domain).

MCP server

StudyDiff's engine is also a Model Context Protocol server, so an agent can ask why two papers disagree as a tool call — the same grounded pipeline, no browser.

Published to the official MCP Registry as io.github.nickjlamb/studydiff, and to npm as studydiff-mcp.

It returns the same auditable report the web app exports: the verdict, the divergent design dimensions (unranked) and the identical ones (ruled out), every value with the verbatim sentence that supports it, the verification counts, and what evidence would resolve the disagreement. Ungrounded fields come back as not reported — never guessed — and the tool never picks a winner, never nominates a primary driver, and never reports a confidence it didn't compute.

Add it to Claude Code:

claude mcp add studydiff -- npx -y studydiff-mcp

…or to Claude Desktop (claude_desktop_config.json):

{  "mcpServers": {    "studydiff": {      "command": "npx",      "args": ["-y", "studydiff-mcp"],      "env": { "ANTHROPIC_API_KEY": "sk-ant-..." }    }  }}

ANTHROPIC_API_KEY is only needed for compare_studies. Leave it out and list_examples / compare_example still work — the cached examples run with no key and no network. (To run from a local clone instead, swap the command for node /absolute/path/to/studydiff/src/mcp.mjs.)

Then just ask:

Use studydiff to compare PMID 19633673 and 20929851 — why do they disagree?

No key handy? list_examples, then compare_example("treg-stability") runs entirely offline.

Project layout

src/ncbi.mjs        PubMed / PMC retrieval (+ DOI resolution, source-depth tagging)src/pdf.mjs         PDF text extraction (pure JS)src/extract.mjs     Claude tool-use → structured study cardssrc/grounding.mjs   deterministic verification (OpenGATE)src/compare.mjs     divergence detection (divergent vs. shared design dimensions)eval/               driver-ranking benchmark: two sets of 15 cited contradictions (dev + held-out) + scorersrc/gaps.mjs        bounded "observed across these papers"src/pipeline.mjs    orchestration: retrieve → extract → verify → comparesrc/report.mjs      shared Markdown report (answer, drivers, quotes, verification)src/server.mjs      web server + streaming API (+ rate limiting, caching)src/mcp.mjs         MCP server: compare_studies / compare_example / list_examplespublic/index.html   single-file dashboard UIfixtures/           cached real papers for the offline demos

Roadmap

Keyword search with a results picker, batch comparison, and comparing more than two papers are the open directions. The benchmark is done and the result held up blind, so the next work is accuracy, not features: better extraction to raise the measured ceiling, and an entailment check to catch prose overstatement that deterministic grounding can't see. Full list — including what's been tried and retired — in ROADMAP.md.

Contributing

Contributions are welcome — see CONTRIBUTING.md for the setup and the invariants that keep the trust guarantee intact.

Provenance

StudyDiff was built for Anthropic's Built with Claude: Life Sciences hackathon (Builder track). All application code in this repository was written from scratch during the event. Grounding uses OpenGATE and PDF extraction uses unpdf, both as published dependencies. Retrieval uses public NCBI E-utilities; extraction uses the Claude API.

License

MIT © Nick Lamb