When a language model's benchmark score shifts after a question is reworded, two quite different explanations are possible. One is that the model is fragile: it reads the problem poorly and small wording changes knock it off course. The other is that the model memorized the question during training, so its score never reflected the ability being tested. Current work blurs the two. Robustness studies treat all sensitivity to wording as fragility. Contamination detectors, the tools that check whether test questions leaked into a model's training data, assume that any sensitivity to wording means memorization. This proposal separates them using a simple behavioral signature. A model that has memorized an item tends to be unusually stable under harmless rewordings, because it recognizes the item. It is unusually wrong in a specific way under rewordings that change the correct answer: it returns the answer to the original question it remembers instead of the question it was asked. We call that second behavior a "stale answer."
We plan to test this under controlled conditions. We inject known benchmark items into the training of open-weight models at controlled exposure levels. We then measure how much of the observed prompt-induced variance comes from contamination and how much from generic fragility, and repeat the measurement under both rigid string-matching and LLM-judge scoring. Finally, we check whether the stale-answer signature carries over to closed API models, where no internal access exists. If it holds, it gives a cheap black-box test that explains, at the level of single items, part of why benchmark rankings are unstable.
Core Claim
Treating "brittleness" as one number hides two effects that point in opposite directions. Brittlebench (Brittlebench) splits score variance into question difficulty and prompt-induced brittleness. It reports that meaning-preserving perturbations cut accuracy by up to 12%, flip pairwise model rankings in 63% of cases, and account for up to half the variance in open models. It does not test whether contaminated items drive any of this. Perturbation-based detectors such as CAP (CAP) and PEARL (PEARL) work in the reverse direction: they take a score drop under perturbation as evidence of memorization.
Our claim is that memorized items have a two-part profile. They show low sensitivity to surface changes such as typos, newlines, personas, and paraphrases that keep the answer. They show a high stale-answer rate under edits that change the answer. Generically fragile items show the reverse surface profile and no excess stale answers. If the claim holds, two things follow. Some of Brittlebench's brittleness term is fragility that contamination would actually reduce. And detectors built on accuracy drops lose signal, because they average the two effects together.
For item $i$, let $u_i$ be the accuracy drop under answer-preserving surface perturbations. Let $r_i$ be the fraction of answer-changing perturbations on which the model outputs the original item's answer, minus the rate expected from a model of equal accuracy that makes errors at random among the options. The proposed memorization score is
$$ m_i = r_i - \lambda\, u_i , $$
where $\lambda \ge 0$ is fit once on the controlled injection data and then frozen. The stale-answer term $r_i$ is the main ingredient. An accuracy drop says only that the model failed. A stale answer says which failure occurred: the model answered a question it was not asked.
Research Program
The work has three parts.
Controlled contamination. We take open models at the scale studied in A Narrowing Geometry in Contaminated Reasoning, namely Qwen2.5-7B and Llama-3.1-8B, and continue training them on ordinary web text from FineWeb. Into that text we inject a random half of the items from ARC-Challenge, MMLU, GSM8K, and GPQA. Each injected item appears at 1, 4, or 16 exposures, in either verbatim or paraphrased form. The other half of each benchmark stays clean. This gives item-level ground truth that no audit of a public model can provide. It also lets us vary exposure to see whether the signature tracks how strongly an item was memorized.
Perturbation families. For every item we generate three families of variants. The first is Brittlebench's surface edits: typos, quoting, newlines, personas. The second is paraphrases written by an LLM and checked by humans to keep the answer. The third is answer-changing edits. For GSM8K these alter the quantities in the problem. For multiple-choice items they shuffle the order of the options, which defines a stale letter, or edit one fact in the question so that a different option becomes correct, as in CAP's logical-consistency edits.
Variance attribution. We fit a mixed-effects decomposition in the style of Brittlebench, adding contamination status and exposure count as factors. This tells us what share of perturbation-induced variance contamination explains, and with what sign. We fit it twice: once under string-match and log-likelihood scoring, and once under LLM-judge scoring. This directly tests the argument in Flaw or Artifact? that most prompt sensitivity is an artifact of the scoring method. That paper reports that Gemma's accuracy standard deviation across ARC-Challenge templates falls from 0.28 to 0.005 under a judge.
We predict that judge scoring removes most of the surface variance on clean items but leaves the stale-answer effect on contaminated items unchanged. A stale answer is semantically wrong, so a correct judge should not give it credit. This prediction would reconcile the two papers. As a secondary white-box check, we compute the Eigenspectrum Concentration Index from A Narrowing Geometry, a measure of how much the model's internal activations collapse into a few directions, on injected and clean items. We also apply its Top-Subspace Suppression method, which removes those dominant directions, to see whether it lowers stale-answer rates. That would tie the behavioral signature to a known internal mechanism.
Transfer to black-box models. We apply $m_i$ with $\lambda$ frozen to API models on the original public benchmarks and on recent LiveBench items released after the models' training cutoffs. Those items serve as a clean reference with low contamination.
Baselines
Do-nothing reference. The naive reference is the plain accuracy drop under paraphrase, the kind of measure behind the roughly 41.5% HumanEval "leak ratio" for ChatGPT that Generalization or Memorization computes against a rephrased HumanEval. We compute it on exactly the same perturbed items as $m_i$. Beating it means a higher item-level AUROC (area under the curve for separating injected from clean items) at every exposure level, and especially at a single exposure, where we expect the drop alone to be weakest.
CDD. The strongest item-level black-box competitor is CDD from the same paper. It samples many completions per prompt and flags an item as leaked when the completions are near-identical to one another. That paper reports relative gains of 21.8% to 30.2% in accuracy and AUC over earlier baselines. We run CDD with its recommended number of samples and give $m_i$ a matched budget of model calls. A narrow AUROC win does not count as beating CDD. Our claim is that $m_i$ matches CDD's AUROC within its confidence interval at equal or lower cost, and that it separates contamination from fragility in the variance attribution, which CDD is not built to do.
CAP. CAP compares accuracy consistency between original and perturbed inputs. It is reimplemented on our benchmarks with its three perturbation types. No reported result was found on Lacuna for CAP's item-level AUROC on general benchmarks; it was evaluated only on financial benchmarks, where it identified 21 leaked FinEval questions. CAP is the closest method, so it doubles as an ablation: it uses the same perturbations as we do, but scores them by accuracy drop instead of stale answers.
ConStat. ConStat detects contamination at the level of a whole model and benchmark. It fits a curve, across a fleet of reference models, that predicts a model's score on one benchmark from its score on a related one, and flags models that beat the prediction. It reports detecting 89% of syntax-contaminated models and 98% of sample-contaminated models, along with estimated inflation of about 8.2% for Mistral-7B on GSM8K and about 6.6% for Llama-3-70B on ARC. We compare at the model level by aggregating $m_i$ across items. Agreement with ConStat's flags on our injected models is a sanity check, not a win condition, because ConStat cannot rank individual items.
Ablations. Three ablations isolate what each part contributes: $r_i$ alone, $u_i$ alone, and $m_i$ computed with random-error correction but without the option-shuffle stale-letter test.
Why not just…?
Why not just switch to an LLM judge, since Flaw or Artifact? shows sensitivity mostly disappears? That result concerns how much accuracy varies across templates, not whether high scores were earned. A judge can make a memorized model look perfectly stable. LiveBench also documents a "judging gap": judges score poorly on problems they cannot solve themselves. Our design treats the scorer as an experimental factor. If judge scoring also removes the stale-answer effect on injected items, then judges are hiding contamination. That result would be more alarming than the original fragility story, and it would be worth reporting.
Why not just run CDD or ConStat and skip the new signature? This is a real threat on detection alone, and CDD may match us on AUROC. Our contribution then rests on attribution: neither method says how much of Brittlebench's ranking instability comes from contamination. ConStat also depends on a fleet of clean reference models, and CDD needs many samples per item. If CDD both matches $m_i$ and explains the same share of variance at equal cost, we will report the stale-answer test as a cheaper equivalent, not as an improvement.
Why not just measure the drop on a paraphrased copy of the benchmark? Under our hypothesis, a paraphrase drop combines the stability that memorization produces with the failures that fragility produces, so the two partly cancel out. The ablation comparing $u_i$ and $r_i$ on injected data tests this directly. If the paraphrase drop alone tracks exposure count as well as $m_i$ does, the two-part claim is wrong and we will say so.
Proposed Experiments and Decisive Checks
The first decisive check is the sign test. On injected items under string-match scoring, the surface drop $u_i$ should be lower than on matched clean items of similar difficulty, while $r_i$ is higher. Both effects should grow with exposure count. If the surface drop is instead higher on contaminated items, the core hypothesis fails and the existing view (memorization is brittle) stands.
The second check is attribution. We report the share of Brittlebench-style brittleness variance explained by contamination status, under each scorer. We also report how often pairwise model rankings flip before and after removing the items with the highest $m_i$, set against Brittlebench's 63% figure.
The third check is black-box transfer. Using the frozen $\lambda$, the share of high-$m_i$ items for API models should be clearly higher on ARC-Challenge, MMLU, and GSM8K than on post-cutoff LiveBench tasks. As a falsification test, it should be near the random-error baseline on LiveBench. A signature that fires equally on fresh LiveBench items is measuring something other than memorization, and the proposal would be judged a failure on that criterion.
References
- Brittlebench: Quantifying LLM robustness via prompt sensitivity
- CAP: Data Contamination Detection via Consistency Amplification
- Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis
- A Narrowing Geometry in Contaminated Reasoning
- Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models