Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng Guo, Pengfei LiuarXiv 2026 · arXiv

TL;DR

Benchmark contamination has evolved from "benchmaxxing" (optimizing for a task distribution) to "benchleaking," where models memorize and recite exact test questions and answers. The authors introduce the Anti-Hack score to adjust for this, revealing that some recent models—most notably ERNIE-5.5-Flash-Base—owe a massive portion of their leaderboard performance to direct memorization of test sets like HumanEval and GSM8K.

What Problem Does This Paper Address?

As LLMs compete for the top spots on leaderboards, the integrity of benchmarks is collapsing. The authors distinguish between two types of contamination:

  1. Benchmaxxing: Optimizing a model to perform well on a benchmark's distribution without seeing the specific test set.
  2. Benchleaking: The actual test examples (questions, options, and reference implementations) enter the training data.

While contamination isn't new, the paper argues it is accelerating and moving "upstream" from post-training (RLHF/Instruction tuning) into the pre-training of base models. This makes these models unreliable foundations for downstream research, as their high scores reflect recall rather than generalizable reasoning capability.

Methods

The researchers employed two primary methods to detect and quantify leakage:

  • Few-shot Continuation: To audit base models, they provided a prompt containing a few solved examples followed by a single test question. They then allowed the model to continue generating text beyond the answer. If the model proceeded to "recite" subsequent test questions, options, and answers that were never in the prompt, it provided direct evidence of benchleaking.
  • Anti-Hack Score: To measure the impact of this leakage, the authors defined a new metric. The Anti-Hack score recomputes a benchmark's total score by marking every detected leaked example as "wrong" while keeping the total number of examples constant.
  • Matching Criteria: They established three levels of matching: L1 (entire question field), L2 (question + options/context), and L3 (question + options + reference answer). The paper primarily uses the strictest L3 criterion for its reporting.

Datasets

The study evaluated four public base models (ERNIE-5.5-Flash-Base, DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base) across 20 benchmarks, including:

  • Commonsense: TriviaQA, HellaSwag, OpenBookQA, PIQA, WinoGrande.
  • Reasoning: MMLU, MMLU-Pro, BBH, DROP, AGIEval.
  • Code: HumanEval, HumanEval+, MBPP, MBPP+.
  • Math: GSM8K, MATH-500.
  • Science: GPQA-Main, SuperGPQA.

Key Insights and Ideas

  • The "Smoking Gun" Typos: In a striking display of memorization, models like ERNIE-5.5-Flash-Base and MiMo-V2.5-Base reproduced test examples including original typos (e.g., writing "examble" instead of "example" in a HumanEval docstring) that exist in the benchmark source but were not provided in the prompt.
  • Contamination is Moving Upstream: The paper documents cases where models recall hidden grading rules during post-training (e.g., Kimi K3.1 Pro recalling specific vocal ranges for a music harmony task that weren't in the instructions). However, the most severe escalation is in pre-training, where base models can recite entire test sets in order.
  • Ranking Reversal: When adjusted for leakage, model rankings change significantly. A model that appears to be a top performer may actually be the weakest once memorized examples are discounted.
Figure 1: Left: given five solved examples and an MMLU abstract-algebra test question, ERNIE-5.5-Flash-Base answers it, then writes out the next test questions in order, with options and correct answers (highlighted); none appears in the input. IDs show zero-based positions in the subject’s test set. The headline percentages refer to ERNIE-5.5-Flash-Base. Right: benchmark averages before and after leaked examples receive zero credit (Anti-Hack). DeepSeek-V4-Flash-Base and Hy3-preview-Base lose 1.5 and 1.3 points; MiMo-V2.5-Base and ERNIE-5.5-Flash-Base lose 5.0 and 16.6.

Key Results

  • Massive Leakage in ERNIE-5.5-Flash-Base: The model leaked 96.3% of HumanEval problems (including reference implementations) and 85.4% of OpenBookQA questions. Its GSM8K leakage was also staggering at 80.1%.
  • Performance Inflation: The Anti-Hack adjustment cost ERNIE-5.5-Flash-Base 16.6 points on average across the suite, moving it from first to last among the four models tested.
  • Cleaner Competitors: In contrast, DeepSeek-V4-Flash-Base and Hy3-preview-Base showed minimal leakage, losing only 1.5 and 1.3 points respectively. This suggests their high scores are largely driven by actual capability rather than memorization.
  • Code and Math are Hotspots: Leakage was most concentrated in coding (HumanEval) and mathematics (GSM8K) benchmarks, which are often the most cited metrics for model "intelligence."
Figure 3: Asked to solve one HumanEval problem, ERNIE-5.5-Flash-Base answers it and then keeps going through the test set in order. The four problems that follow were not in the input in any form, and each is reproduced word for word with its task description and reference implementation—with fruit_distribution shown in full (highlighted), down to the typo examble in the original. On the left, the first demonstration and the question are given in full while the other demonstrations appear as signatures; on the right, the problems not shown in full are listed by function name and test-set ID.

Limitations and Future Work

The authors acknowledge that their strict matching (L3) is a conservative estimate. It likely misses "paraphrased" leakage where a model has seen the data but reproduces it with slight variations in wording or code structure. Furthermore, a "zero" leakage score in this study does not prove a model is uncontaminated; it only means the specific few-shot continuation probe did not trigger a recitation.

The paper concludes with a call for the AI community to make decontamination reports and contamination-adjusted scores a standard requirement for all new model releases to ensure scientific progress is measured against actual reasoning, not just memory.

Summary updated

Abstract

Benchmark contamination is not new, but recent model releases show it in a sharper form. We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing, which optimizes toward a benchmark without seeing its test set. We document two escalations over the past month. First, in post-training, the traces have become explicit: recent audits show an agent stating hidden test requirements it was never given in 23% of its attempts, where its predecessor did so in none, and continuing to follow them after the task is changed to contradict them (Schaeffer and Räuker, 2026; Lo and Lo, 2026). Second, the problem has reached pre-training. Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input. For the recently released ERNIE-5.5-Flash-Base, 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base. To quantify what a reported score owes to such examples, we define an Anti-Hack score: the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged. Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base. Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.
BibTeX
@misc{qin2026benchleaking,
  author = {Yiwei Qin and Muhang Xie and Xuefeng Li and Mohan Jiang and Dayuan Fu and Tiantian Mi and Qipeng Guo and Pengfei Liu},
  title = {{Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models}},
  year = {2026},
  eprint = {2611.21315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21315},
}
Figure