TL;DR
Benchmark contamination has evolved from "benchmaxxing" (optimizing for a task distribution) to "benchleaking," where models memorize and recite exact test questions and answers. The authors introduce the Anti-Hack score to adjust for this, revealing that some recent models—most notably ERNIE-5.5-Flash-Base—owe a massive portion of their leaderboard performance to direct memorization of test sets like HumanEval and GSM8K.
What Problem Does This Paper Address?
As LLMs compete for the top spots on leaderboards, the integrity of benchmarks is collapsing. The authors distinguish between two types of contamination:
- Benchmaxxing: Optimizing a model to perform well on a benchmark's distribution without seeing the specific test set.
- Benchleaking: The actual test examples (questions, options, and reference implementations) enter the training data.
While contamination isn't new, the paper argues it is accelerating and moving "upstream" from post-training (RLHF/Instruction tuning) into the pre-training of base models. This makes these models unreliable foundations for downstream research, as their high scores reflect recall rather than generalizable reasoning capability.
Methods
The researchers employed two primary methods to detect and quantify leakage:
- Few-shot Continuation: To audit base models, they provided a prompt containing a few solved examples followed by a single test question. They then allowed the model to continue generating text beyond the answer. If the model proceeded to "recite" subsequent test questions, options, and answers that were never in the prompt, it provided direct evidence of benchleaking.
- Anti-Hack Score: To measure the impact of this leakage, the authors defined a new metric. The Anti-Hack score recomputes a benchmark's total score by marking every detected leaked example as "wrong" while keeping the total number of examples constant.
- Matching Criteria: They established three levels of matching: L1 (entire question field), L2 (question + options/context), and L3 (question + options + reference answer). The paper primarily uses the strictest L3 criterion for its reporting.
Datasets
The study evaluated four public base models (ERNIE-5.5-Flash-Base, DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base) across 20 benchmarks, including:
- Commonsense: TriviaQA, HellaSwag, OpenBookQA, PIQA, WinoGrande.
- Reasoning: MMLU, MMLU-Pro, BBH, DROP, AGIEval.
- Code: HumanEval, HumanEval+, MBPP, MBPP+.
- Math: GSM8K, MATH-500.
- Science: GPQA-Main, SuperGPQA.
Key Insights and Ideas
- The "Smoking Gun" Typos: In a striking display of memorization, models like ERNIE-5.5-Flash-Base and MiMo-V2.5-Base reproduced test examples including original typos (e.g., writing "examble" instead of "example" in a HumanEval docstring) that exist in the benchmark source but were not provided in the prompt.
- Contamination is Moving Upstream: The paper documents cases where models recall hidden grading rules during post-training (e.g., Kimi K3.1 Pro recalling specific vocal ranges for a music harmony task that weren't in the instructions). However, the most severe escalation is in pre-training, where base models can recite entire test sets in order.
- Ranking Reversal: When adjusted for leakage, model rankings change significantly. A model that appears to be a top performer may actually be the weakest once memorized examples are discounted.
Key Results
- Massive Leakage in ERNIE-5.5-Flash-Base: The model leaked 96.3% of HumanEval problems (including reference implementations) and 85.4% of OpenBookQA questions. Its GSM8K leakage was also staggering at 80.1%.
- Performance Inflation: The Anti-Hack adjustment cost ERNIE-5.5-Flash-Base 16.6 points on average across the suite, moving it from first to last among the four models tested.
- Cleaner Competitors: In contrast, DeepSeek-V4-Flash-Base and Hy3-preview-Base showed minimal leakage, losing only 1.5 and 1.3 points respectively. This suggests their high scores are largely driven by actual capability rather than memorization.
- Code and Math are Hotspots: Leakage was most concentrated in coding (HumanEval) and mathematics (GSM8K) benchmarks, which are often the most cited metrics for model "intelligence."
Limitations and Future Work
The authors acknowledge that their strict matching (L3) is a conservative estimate. It likely misses "paraphrased" leakage where a model has seen the data but reproduces it with slight variations in wording or code structure. Furthermore, a "zero" leakage score in this study does not prove a model is uncontaminated; it only means the specific few-shot continuation probe did not trigger a recitation.
The paper concludes with a call for the AI community to make decontamination reports and contamination-adjusted scores a standard requirement for all new model releases to ensure scientific progress is measured against actual reasoning, not just memory.
Summary updated