Keystone

Keystone

[Live site] [Case study] [MCP registry] [License]

What is this paper standing on?

Live: https://keystone-research.vercel.app

No paper stands by itself. Each one adopts a method from somebody, argues with somebody else, and takes a handful of things on faith without flagging them. Keystone reads all three out of the paper's own LaTeX — not a summary, not a retrieval index, no model in the loop — and pins each one to the pixels on the page where the paper says it.

The name is the thesis, and it took a pivot to earn it. A keystone is the one stone an arch collapses without, and for almost every paper that stone belongs to somebody else.

Three things, all quoted rather than inferred:

• What it stands on. A citation is not a neutral pointer. "Following \cite{ba2016layer} we normalise each layer" and "Unlike \cite{ba2016layer}, we normalise across the batch" are opposite statements about the same paper, and the difference is written down. Every citation site is classified from its cue phrase, and the cue is shown, so the reading is something you can disagree with.
• What it takes on faith. Authors announce their assumptions because the genre requires it — "we hypothesize", "for simplicity", "it is well known that". Each is quoted next to what the paper offers in the same sentence: a citation, a pointer to its own evidence, or nothing at all.
• The graph. Where a cited paper is also in the library the edge is walkable in both directions, so a paper shows both what it stands on and who stands on it. Across 234 papers that is 548 links, each carrying the sentence that placed it, and the chain is the real one: VGG → ResNet → Transformer → BERT / ViT / LLaMA, with batch and layer normalisation feeding in.

The library grows along its own citations. keystone expand reads every reference the library already resolves to arXiv, ranks them by how many papers take a stance on each, and fetches the top of that list — so each new paper arrives with at least one edge already attached rather than as an isolated node.

ResNet's central hypothesis — that residual mappings are easier to optimise than unreferenced ones — is stated once, in the introduction, with nothing offered for it. Page 2.

And it is often silent, and sometimes wrong. A cue read out of position produces a confident opposite rather than a vague answer, so the error rate is published rather than implied — and measured on somebody else's labels, because a rate measured on the same papers the rules were written against is not a rate, it is a restatement.

Scored against SciCite (11,014 citation contexts, Cohan et al. 2019) and ACL-ARC (1,941, Jurgens et al. 2018), neither of which contributed a single cue phrase:

Both numbers matter. Keystone is built to refuse: no cue phrase written down, no stance reported, nothing shown. On a broad corpus that means it declines most citations — "using CMet cardiac metric software [12]" is a real adoption whose only evidence is knowing what CMet is. Its macro-F1 is 0.325 and 0.204, below the 0.273 a cue-phrase system from 2000 scores on ACL-ARC and far below the 0.84–0.889 of trained neural classifiers, because macro-F1 punishes silence exactly as hard as error. That figure is published too; leaving it out would be the convenient choice.

On 90 citations from this library labelled by hand, accuracy is 87% [78–92%] and macro-F1 0.84, against 0.59 for a bag-of-words classifier over the same sentences. That sample says the rules are self-consistent, not that they travel — see eval/corpora/README.md, which also records the three real defects scoring against public labels found and fixed. Roughly one reading in five is wrong either way, which is why every row shows the sentence and the cue it keyed on.

Why not audit the arithmetic? That was the first version, and it shipped. A paper's numbers can only be reported as an absence of errors, and careful papers have none: across nine foundational papers the check suite correctly produced zero findings, so its best possible output was a blank page. The numeric analysis is still here — claims traced to table cells, every number indexed, baselines checked against the papers they cite — but below the lineage, where a detail belongs. See RELEASE.md for the pivot.

Landscape

Two things are true and worth stating plainly.

The mainstream tools — SciSpace, ChatPDF, Elicit, Paperguide, NotebookLM — are all the same pipeline: chunk, embed, retrieve, answer, cite a page number. They compete on corpus size and polish, not capability.

But Keystone is not the first tool to pin paper criticism to the page. Loupe (Apache 2.0) reviews scientific papers in a cheap triage pass followed by an opt-in deep dive, and pins every finding to a bounding box verified by a vision model. Someone converging independently on pixel-pinned findings is evidence the direction is right. Where Keystone differs:

Loupe's issue taxonomy (logic, quantifier scope, missing step) is oriented at proofs. Keystone's is oriented at experiments, which is where ML/CS papers actually go wrong.

Status

Live at https://keystone-research.vercel.app. The numbers below are read from the generated JSON the build writes, so they cannot drift from the run that produced them.

The anchor layer. Tight anchors survive an independent geometric round-trip through MuPDF rather than through the matcher's own index; the gate is ≥99% and the suite refuses to build below it. Real spans are never rejected and fabricated quotes are never anchored — both are gated at zero. Roughly 93% of a paper's LaTeX sentences can be found in its PDF, and a source that matches its PDF on fewer than a quarter of sampled sentences is refused outright rather than published.

The cue rules, scored on somebody else's labels. Against SciCite (11,014 citation contexts) the rules report a stance for 4% of citations and are right about 80% of those, 95% CI 76–84%. Macro-F1 is 0.325 against neural classifiers near 0.84, and that gap is the deliberate trade: macro-F1 punishes silence exactly as hard as error, and this is built to be silent where a sentence carries no cue. See /method on the site.

What runs. The reader with pixel-anchored highlights, highlights and notes for signed-in readers, Google sign-in with cross-device sync, search and a library guide, library-wide views (/timeline, /contested, /assumptions), a feed of new work (/news), on-demand analysis of any arXiv paper that ships LaTeX source, an MCP server at /api/mcp, and cookieless analytics.

Deliberately not built. Per-visitor tracking, and any claim the parser cannot anchor. The deterministic numeric checks were largely retired: on carefully reviewed papers their best case was a blank page, and the project's value turned out to be lineage rather than arithmetic. See RELEASE.md.

Why the anchor layer got this much attention

Every downstream claim depends on it. A finding that says "the abstract reports 94.2% but Table 3 says 93.8%" is worthless unless the reader can click it and land on both numbers. Getting there took several rounds of measurement, and the interesting part was that the first three failures were in the test, not the code:

• Round-trip started at 67.6%. Cause: a tall ( from a display formula legitimately clips into a body line, breaking exact string containment.
• Union bounding boxes let one tall glyph (→∞) stretch a line's highlight over the whole of the next line. Real defect; fixed with median line boxes.
• Content-stream word order and clipped-text order genuinely differ across columns and table rows, so any whole-span ordering assertion is invalid. Now checked per rectangle, where order is unambiguous — a stricter test, not a looser one.
• Fuzzy matching accepted 5.9% of fabricated quotes while matching zero of 1846 real spans. partial_ratio scores the best window, so a single substituted word barely moves it: "Trade, feature 9-50" scored 93.6 against "Trade, pages 9-50". For a tool whose job is verifying what a paper says, that is the worst possible failure mode. Fuzzy is now opt-in, with a word-substitution guard.

Anchors self-report precision (tight / approximate), and verification holds each to its own contract. In stacked geometry — a fraction's numerator over its denominator — no axis-aligned rectangle can be tight, so an anchor says so instead of implying a precision it does not have.

What the checks taught us

Every false positive the suite has ever produced was a trigger error, not an arithmetic error — which is the specific danger of deterministic checking. A check that computes correctly but fires on the wrong premise asserts nonsense with certainty, and is worse than a model's hedged guess. The four that reached real papers:

• Substring label matching. "sum" in "summarization" and "avg" in "avg pool" turned a method name and a layer name into aggregate rows, producing 21 high-severity findings on correct tables. Labels are now matched whole.
• Assumed orientation. Bold marks the best down a column in some tables and across a row in others. Assuming the first produced 31 confident findings on one correct InstructGPT table. Orientation is now inferred per table and validated.
• Ignored band structure. A results table is banded — prior work, a rule, then the authors' rows — and "best" is a claim about a band. Judging the Transformer's Table 2 down the whole column reported the best ensemble as a middling value. \midrule is now kept as structure rather than stripped as formatting.
• Comparing unlike quantities. A ResNet row holding [email protected] and mAP@[.5,.95] for two datasets was compared end to end, reporting the smaller metric as failing to be best. Row comparisons now group by the innermost column header when those headers repeat.

The common remedy is the same in each case: a check must validate its own premise before stating its conclusion. A total row is only reported as wrong if it adds up correctly in some other column; emphasis is only read as "best" if that reading explains the paper's other markers.

What limits recall is not the arithmetic but the frame: whether a table compares down columns or across rows, and which rows are peers. That is exactly the kind of judgement the plan assigns to the model — extraction and routing — while the verdict stays arithmetic.

Layout

services/core/keystone/ingest/├── normalize.py      skeleton projection: ligatures, accents, dashes, hyphenation├── pdf.py            MuPDF word geometry — the ground truth├── anchors.py        quote -> rectangles, with the rejection path├── verify.py         independent round-trip through MuPDF, per precision contract├── regions.py        stacked geometry, rotated lines, maths classification├── furniture.py      running heads, page numbers, the rotated arXiv stamp├── latex.py          arXiv source: equations, tables, citations, sentence runs├── arxiv_source.py   e-print fetch, unpack, main-file resolution├── tables.py         LaTeX tabular -> typed cells, with emphasis and bands├── assemble.py       ingested sources -> Paper└── sampling.py       span sampling for the anchor audit
services/core/keystone/├── lineage/stance.py       the cue rules: adopts / extends / disputes / compares├── lineage/assumptions.py  what a paper takes on faith, and what it offers for it├── lineage/graph.py        paper-to-paper edges, for the library views├── audit/numbers.py        number parsing that preserves reported precision├── audit/registry.py       check registration and the runner├── audit/checks/tables.py  table consistency checks├── graph/models.py         the typed structures checks reason over├── eval/external.py        scoring the cue rules on SciCite and ACL-ARC└── eval/planted.py         planting known defects to measure detection

The reader and everything served from it:

apps/web/├── app/                    routes: the reader, the library views, /news, /stats├── app/api/                account, marks, chat, guide, feed, mcp, track├── app/api/ingest.py       Python function: analyse any arXiv paper on request├── components/             the reader, the page view, the graph, the timeline├── lib/                    marks geometry, anchors, analytics, feeds, the store├── test/                   node --test, no framework└── public/dossiers/        the built library, committed so the site needs no database

Running it

cd services/core && uv sync

Fetch the test corpus (ten public arXiv PDFs):

./eval/fetch_corpus.sh

Run the gates:

cd services/core && uv run pytest -q

Audit anchor accuracy across the corpus:

cd services/core && uv run keystone audit ../../eval/corpus/pdf/*.pdf --samples 150 --hyphen-samples 40

Measure how much arXiv LaTeX source anchors back into each PDF:

cd services/core && uv run keystone source-audit ../../eval/corpus/pdf/*.pdf

Run the deterministic checks, and measure them against planted defects:

cd services/core && uv run keystone check 1706.03762 1810.04805

cd services/core && uv run keystone planted-audit 1706.03762 1512.03385 1810.04805 --verbose

Inspect one quote, and render its highlight to an image to eyeball:

cd services/core && uv run keystone probe ../../eval/corpus/pdf/1706.03762.pdf "The Transformer allows for significantly more parallelization"

The arXiv source path

For any arXiv paper the actual LaTeX is available at arxiv.org/e-print/{id}, which gives exact equations, table numbers as the author wrote them, and real \cite keys — none of which are reliably recoverable from rendered glyphs. Source-derived text is then anchored back into the PDF so those facts still have pixel coordinates.

Sentence-level alignment reached 93.5% through four fixes, each found by measurement: expanding argument-less user macros (\etal, a method shorthand) that otherwise leave holes where the PDF shows text; brace-matched removal of macro definitions, since nested braces defeat a regex; excluding page furniture; and splitting runs into sentences.

Submissions that wrap a pre-compiled PDF (\includepdf) raise SourceUnavailable rather than returning an empty document — reporting "no equations, no tables" would be a false statement about the paper rather than an absence of information about it.