Audit a validation for leakage: does outside ground-truth actually enter, or is everyone confirming a result no one independently produced?
Goal
The name is the Mandela Effect — a whole population confidently remembers something that never independently happened; a leaky validation is the same shape. Walk the 8 patterns below. mandela checks one thing: whether a validation is independent, or whether the designer, model, and scorer are only confirming each other.
Workflow
- Identify the validation (eval / metric / experiment / holdout / "how we'll know"). Name its components — what plays model, scorer, designer, dataset.
- Ask the core question: does external ground-truth enter independently?
- Test the validation against all 8 patterns below (some apply only to certain components — a human subject, a scorer); report only the ones that fire, each by name.
- Give the independent-ground-truth fix for each hit.
The 8 leakage patterns
- Recall, not reason — a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
- Wrong null hypothesis — an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
- Shared hallucination — two components verifying each other; circularity reported as a number.
- Tautology — a scorer grading buckets it drew itself.
- Verifier = designer — a private, unreproducible recipe in a holdout's clothes.
- Shared-pool bias — train and holdout drawn from one labeler pool, so one bias enters both sides.
- Frame injection — a question that hands the subject the hypothesis.
- Demand characteristics — measured subjects who know they're being measured.
Rules
- Subtlety that bites twice: you can blind the output value and still leak the collection recipe.
- Read-only — name the leak and the independence fix; don't rewrite the experiment.
- For a high-stakes validation, you may add one independent fresh-context auditor (
N=1) handed only the validation design, not this session's reasoning, to re-run the 8-pattern taxonomy blind; the default remains same-session and read-only.
Verification
Turn mandela on this audit:
- Run patterns #3–#5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? is the verifier the designer (#5)? is your verdict a shared hallucination with the design's own claims (#3)?
- Could a reader who didn't run the audit reach your verdict from the cited evidence alone — independent ground-truth?
- The report names the root, not a laundry list.
