Detecting Cloaked Benchmark Contamination by Compressing Correct Answers

When a language model has trained on a benchmark's test questions, its score stops measuring reasoning and starts measuring recall. The tools for catching this mostly ask how confident the model is on the test text, for example how low its loss is or how improbable its least likely tokens are. Recent work shows these confidence signals can be erased. Reinforcement-learning post-training on clean data, and contamination through worked solutions instead of verbatim text, both drive ten detectors to near-chance accuracy, yet the inflated scores remain (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Something item-specific must survive, because the score boost survives. We propose to look for it in what the model writes, not in how confident it is.

The idea in two sentences. Take a model's correct sampled solutions to a test item. Replace every number and named entity with a typed placeholder, so that whatever a genuine solver is forced to reproduce no longer counts. Then measure how many bits the item's reference solution saves when compressing what is left, compared with the same measurement on fresh items from the same distribution. A model that solved the problem itself should share only the parts of the solution that the problem determines. A model that learned the solution should also reuse the parts that were free choices: the order of steps, which quantity it computes first, the wording of the justification. Compression-based model selection supplies a way to count that excess reuse without picking a fragment length or a similarity threshold.

Core Claim

We claim that contamination which survives reinforcement-learning "cloaking" and chain-of-thought training shows up as excess, non-determined reuse of solution structure in correct outputs. We also claim that a two-part minimum description length (MDL) code measures this reuse more reliably than surface overlap scores. MDL is the principle that the best description of data is the one giving the shortest total encoding: the cost of the model plus the cost of the data given the model.

Concretely, let $\tilde S_i$ be the placeholder-normalized correct samples for item $i$, $q_i$ its question, $R_i$ its normalized reference solution, and $B$ a background code table. A code table is a dictionary of recurring token fragments with code lengths that shrink as usage grows. $B$ is selected by MDL from the same model's normalized outputs on a fresh pool of items. The item score is

$$g_i = \frac{1}{|\tilde S_i|}\Big[L(\tilde S_i \mid B, q_i) - L(\tilde S_i \mid B, q_i, R_i)\Big],$$

where $L(\cdot)$ is the two-part description length in bits. Conditioning on $R_i$ lets the coder reference fragments of the reference solution at pointer cost, and conditioning on $q_i$ makes copying from the question nearly free. Fragments enter the cover only if they pay for their own description. Generic phrasing is therefore absorbed by $B$, and fragments that merely restate the question cost nothing. We compare $g_i$ against the distribution of the same statistic on fresh items with their own references, within bins of the model's accuracy on those items, and use the resulting z-score as the contamination score.

Relation to Prior Work

The method comes from the pattern-mining line of work on compression. Krimp selects a small set of itemsets by asking which set compresses a database best, reducing billions of candidates to a few hundred. SQS and Seqret carry the same idea to event sequences and rules. We reuse their greedy cover-and-prune selection over token sequences. Our novelty is the conditional, background-subtracted comparison, not the search procedure.

DUPEX compresses legal boilerplate after replacing dates and amounts with typed placeholders. Its open problem was that placeholders are a manual workaround for variation. We use the same device on purpose, as a way to delete exactly the content that a genuine solver would reproduce. Compressive Feature Learning shows that dictionary MDL over text can be made order-invariant and scalable, but it caps patterns at five tokens. Greedy covers do not need that cap, and we keep it open as a scaling fallback. MIDGARD uses MDL to merge several sampled reasoning graphs into one consensus graph. We likewise operate on sets of samples, but for auditing rather than decoding.

On the contamination side, Rethinking LLM Memorization through the Lens of Adversarial Compression is the closest compression-framed work. It defines memorization as the ratio of a target string's length to the shortest prompt that elicits it, found by hours of gradient-guided prompt search per string, and it studies verbatim text such as quotes and books. Our approach differs in three ways. We study reasoning benchmarks rather than verbatim text. We measure structure after normalization rather than exact strings. Our cost is a handful of samples plus a CPU-side cover, not an optimization per item.

Several papers motivate the gap. Likelihood-based membership tests such as Min-K%++ score training-set exposure from normalized token log-probabilities. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows these collapse under reinforcement learning and chain-of-thought contamination. Evading Data Contamination Detection for Language Models is (too) Easy and Rethinking Benchmark and Contamination for Language Models with Rephrased Samples show that rephrasing defeats surface matching.

The same fragility paper also constrains us. It reports that reasoning models "internalize reasoning structures rather than surface tokens." That is why we measure structure after normalization. If the internalized structure transfers equally to unseen items, no item-level detector can succeed, and we say so in the falsification checks below.

Benchmark-level comparisons, namely ConStat, GSM1k, and MMLU-CF, tell you whether a score is inflated. They need a fleet of reference models or a new benchmark, and they do not say which items are affected. Our score is item-level and needs only the audited model and a pool of fresh items.

Research Program

The pipeline has four parts.

First, sampling. For each item we draw $k=8$ chain-of-thought solutions at a fixed temperature and keep the correct ones. Items with no correct sample receive no score, and we report coverage.

Second, normalization. Numbers, named entities, and answer-option letters become typed placeholders, using DUPEX-style rules extended with a numeric-expression tagger. This also covers calculator-style annotations of the form "48/2=24".

Third, coding. We implement a Krimp/SQS-style greedy cover over token sequences with no length cap. Within it, the three code tables (background $B$, question $q_i$, reference $R_i$) compete for each span under a single two-part code. The background $B$ is mined once per model from normalized outputs on the fresh pool.

Fourth, calibration. We z-score $g_i$ against fresh items within accuracy bins. The reported number is item-level, and a benchmark-level estimate is formed by summing over items.

We also build a reference-free variant for audits in which the contaminating traces are unknown. In this variant $R_i$ is replaced by the other samples of the same item, so the question becomes how much of the model's solution set is explained by its own item-specific repeated fragments beyond the background. This variant is more exposed to a confound: reinforcement learning sharpens every output distribution and therefore makes all items more self-consistent. We treat it as secondary and test it explicitly.

Baselines

The naive reference point is benchmark accuracy compared with accuracy on a matched fresh set. This is the GSM8K versus GSM1k comparison, or MMLU versus MMLU-CF validation. It costs nothing beyond evaluation and gives only a benchmark-level gap. GSM1k reports drops of up to 8% for contaminated model lineages, and MMLU-CF reports an average drop of about 14% across 40 models. We beat this baseline if our summed item scores recover the injected contamination fraction and also rank the contaminated items correctly, which the gap cannot do.

The strongest existing item-level methods are the likelihood detectors evaluated in the fragility study: loss, Min-K%++, and the likelihood-ratio attack LiRA. These are the methods the field actually uses, and the fragility paper gives reported numbers in our exact setting. After contamination by supervised fine-tuning, LiRA reaches 89.1% AUROC. One epoch of clean GRPO cuts LiRA by about 14% and loss by about 16%, pushing reference-free methods to about 50% AUROC. Under chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B, the ten detectors average 50–58% AUROC (On The Fragility of Benchmark Contamination Detection in Reasoning Models). We also report the evasion result that rephrased training pushes sample-level detection to about 1% true-positive rate at 1% false-positive rate (Evading Data Contamination Detection for Language Models is (too) Easy).

We rerun all of these ourselves on the same models, items, and member/non-member splits. LiRA gets the same number of reference models it used in the original study, and every method gets equal hyperparameter search on a validation split. Beating them means AUROC of at least 70% in the post-GRPO and chain-of-thought conditions, together with a TPR at 1% FPR that is clearly above 1%. Both must hold over bootstrap confidence intervals across three contamination seeds.

The adversarial compression ratio (Rethinking LLM Memorization through the Lens of Adversarial Compression) is the closest compression-based competitor. It is too expensive to run on full benchmarks, so we run it on a random 200-item subset per condition, with a matched GPU budget reported alongside. The comparison is accuracy on that subset and accuracy per GPU-hour.

Three ablations isolate what is new. The first removes the background table, so raw reference savings are counted. The second removes placeholder normalization, so determined content counts as well. The third keeps everything except the MDL cover and substitutes ROUGE-L between normalized outputs and normalized reference, with background-frequent n-grams removed. This last ablation is the decisive test of whether MDL itself earns its place.

Why not just…?

Why not just measure n-gram or ROUGE overlap between the model's answer and the reference solution? This is the most serious threat. Raw overlap is confounded by determined content, since any correct solver reproduces the intermediate values. Overlap has also already failed against rephrased contamination (Rethinking Benchmark and Contamination for Language Models with Rephrased Samples). However, once normalization, correctness-conditioning, and fresh-item calibration are added, a background-filtered ROUGE could come close to our score. That is why it is our third ablation. MDL should help in two places. It counts overlapping fragments once instead of repeatedly, and it charges each fragment a model cost, so short coincidental matches contribute nothing. If the ablation matches us within confidence intervals, we will report that the contribution is the measurement design rather than the compressor, and recommend the cheaper scorer.

Why not just check the model's confidence on the reference solution, using teacher-forced loss or Min-K%++? That is precisely the signal shown to vanish after clean GRPO and under chain-of-thought contamination (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Our hypothesis is that the clipped policy updates which flatten log-probability gaps do not remove item-specific choices in the model's outputs, because the score inflation persists. The Stage I and Stage II comparisons test this directly.

Why not just measure self-consistency across samples, such as self-BLEU or answer entropy? For the reference-free variant this is a real competitor. Reinforcement learning makes all items more self-consistent, and a plain diversity score cannot separate "every item uses the same template" from "this item has its own repeated fragment." Our background table is designed to make exactly that separation. We run self-BLEU and sample entropy, both calibrated by accuracy bin, against the reference-free variant. If they match it, we drop the reference-free claim.

Why not just check whether answers to number-perturbed versions of an item still contain the original item's values? For arithmetic benchmarks this is a strong, cheap probe, and we include it as a baseline on GSM8K. It does not extend to multiple-choice knowledge benchmarks like MMLU, where there are no values to perturb. Our score applies to both kinds of benchmark.

Proposed Experiments

All experiments use controlled contamination, so that ground truth is known. On GSM8K, MMLU (with chain-of-thought explanations), and the competition-math set used by the fragility study (AIME, AMC, OlympiadBench, Minerva Math), we randomly split each benchmark into member and non-member halves. The fresh pool for the background table and for calibration is GSM1k if it has been released and a held-out GSM8K split otherwise, MMLU-CF's open validation set, and a held-out part of the math pool. A base model of about 8B parameters and DeepSeek-R1-Distill-Llama-8B are fine-tuned on the members under four conditions: verbatim question–answer pairs, evasively rephrased pairs following Evading Data Contamination Detection for Language Models is (too) Easy, chain-of-thought traces distilled from a stronger model, and each of the previous three followed by one epoch of clean GRPO. Metrics are item-level AUROC and TPR at 1% FPR, benchmark-level error in the estimated contamination fraction, and wall-clock cost.

Three checks decide the outcome.

First, inflation without item specificity. Before scoring, we measure whether members gained more accuracy than non-members. If they did not, the contamination generalized within the distribution, item-level detection is impossible in principle, and that condition is reported only at benchmark level.

Second, the cloaking test. Where members did gain more, the proposal succeeds only if $g_i$ stays above 70% AUROC after GRPO, the condition in which likelihood detectors reach about 50%.

Third, the instrument test. The full MDL score must beat the background-filtered ROUGE ablation, and the reference-free variant must beat calibrated self-BLEU. If either comparison fails, we report the failure as the finding.

We also run a hidden-trace audit. The auditor holds only the public reference solutions while the model was trained on distilled traces. This shows how much of the signal survives when the contaminating text is unknown, which is the realistic setting for third-party audits.