Contamination as a Differential Test: Peer-Calibrated Twin Gaps

A model that has memorized a benchmark question should do noticeably better on that exact question than on a freshly written "twin": the same problem with different wording, numbers, or distractors. Several contamination detectors use this idea. They face two confounds that make their verdicts unreliable. First, twins are rarely exactly as easy as the originals. Second, a twin written by a language model reads differently from the human original, so models may prefer the original simply because it is more natural text. We propose to borrow the central move of differential testing, the software-testing practice of feeding the same input to several independent implementations and treating disagreement as the bug signal. Applied here, a target model's original-versus-twin gap is judged against a panel of peer models, including anchors whose training data is public and can be checked. We use machine-generated-text detectors in a narrow role: they measure how "machine-like" each twin is, and that measurement is removed as a covariate. What remains is the part of the gap that peers of similar ability do not show. That excess is our contamination estimate, and it needs only text outputs.

The opportunity is concrete. A controlled study of five contamination detectors found near-zero agreement between methods on the same model and dataset pairs. It also found that contamination introduced through instruction fine-tuning with augmented answers defeats Min-K% Prob entirely (Towards Data Contamination Detection for Modern Large Language Models). Most accurate detectors also require token log-probabilities, which closed APIs increasingly hide. A black-box test whose decision rule is calibrated against peers, rather than against an assumed-fair perturbation, would address both problems.

Core Claim

The original-minus-twin accuracy gap is a contamination signal only after removing two nuisance effects. One is twin difficulty, which depends on model capability. The other is the fluency difference between a human-written original and a machine-written twin. A panel of peer models estimates the first, and text-detection statistics estimate the second. For model $m$ and item $i$, let $g_{m,i}$ be the model's accuracy on the original minus its mean accuracy over $K$ twins. We fit a predictor on the panel and define the contamination excess

$$\kappa_{m,i} = g_{m,i} - \hat g(\theta_m, d_i, f_i),$$

where $\theta_m$ is the model's capability (its overall accuracy on twins only, which cannot be memorized), $d_i$ is the item's panel-estimated twin difficulty shift, $f_i$ is the fluency gap between original and twins measured by a detector, and $\hat g$ is the panel's predicted gap for a clean model of that capability. Benchmark-level contamination is the mean of $\kappa_{m,i}$, tested against a permutation null obtained by swapping original and twin labels. Item-level scores come from averaging over the $K$ twins.

The claim is that this excess separates contaminated from clean models where single-model gaps do not. It should also hold under instruction fine-tuning with chain-of-thought augmentation, the setting where likelihood detectors fail.

Relation to Prior Work

The closest work applies semantics-preserving perturbations within a single model. PaCoST rephrases questions with a helper model and runs a paired t-test on the target's probability of "Yes". This requires log-probabilities and implicitly assumes rephrased items are equally answerable. CAP shuffles options, alters distractors, and paraphrases, then compares a consistency ratio between training and test splits. It is black-box, but it needs a training split for calibration and was shown mainly on financial benchmarks. Both supply our perturbation machinery. Our proposal differs by replacing their implicit oracle ("a clean model would be equally consistent") with an empirical one estimated from peers. This is the move that differential testing of JavaScript engines makes when majority behavior across independent engines stands in for a missing specification.

Single-model internal detectors play a contrasting role. LOS-Net trains a small transformer on top-K output distributions. AECA filters per-token loss and entropy as a signal. RADAR classifies 37 internal features. All three need gray- or white-box access, and AECA reports degradation on short sequences, which is exactly the length of most benchmark items. We serve as a black-box complement, and we use them as gray-box comparison points on open models.

Machine-generated-text detection provides the fluency control, and its role in this proposal is motivated by a specific failure. The inconsistency study's "Fluency Trap" shows that models pick original wording over synonym-substituted text because the original is more natural, not because it was memorized. DetectLLM gives a single-pass log-likelihood-to-log-rank ratio (LRR) computed under an open scoring model. We use the LRR difference between original and twin as $f_i$, and we also use it to filter or rewrite twins until they fall within a fluency band of the original.

We deliberately do not rely on persistent-homology intrinsic dimension (Tulchinskii et al.) for this covariate. Like other statistical detectors, it needs more token mass than a benchmark item provides, as both MGTBench and AECA document for short text. This is a constraint on our design, not a source of method.

The method has one direct weakness, and we inherit it from differential testing: shared failure. If every peer saw the same leaked benchmark in a common web crawl, majority voting is blind. This is the "collusion failure" of the field. We address it with anchors, as described next, and we report only excess contamination relative to the panel when no anchor is clean.

Research Program

We will build three components and test their interaction.

The first is a twin generator. For formal tasks, DyVal produces structurally matched problems from the same graph, with known difficulty. For natural-language benchmarks such as MMLU, GSM8K, and FinEval, an open model rewrites each item with CAP-style factual, logical, and semantic perturbations, producing $K$ twins per item. We keep only twins whose LRR falls within a band around the original.

The second is a peer panel of roughly ten open and closed models spanning a wide range of capability. It includes anchors whose contamination status is known. One kind of anchor is a model with public training data, such as Pythia on The Pile, where n-gram audits are possible. The other kind is a set of controlled fine-tunes of a single open base. We fine-tune that base on half of a benchmark, with or without chain-of-thought answer augmentation, following the Oracle protocol of the inconsistency study, and keep the other half as clean ground truth.

The third is the estimator $\hat g$. It is fit only on models and items currently believed clean, and it is refit with leave-one-model-out so that a target never calibrates against itself.

The main analyses ask three questions. Does $\kappa$ recover injected contamination, both verbatim and chain-of-thought-augmented, at item and benchmark level? Does it agree with known real cases? Does removing the panel or the fluency covariate destroy that recovery?

Baselines

The naive reference point is the raw original-minus-twin accuracy gap for a single model, flagged when it exceeds zero by a paired test. This is the "do nothing beyond perturbation" detector. If our method cannot beat it, the panel adds nothing. A second naive reference is n-gram completion overlap, which the inconsistency study found returns almost no matches on frontier models.

The strongest black-box competitor is CAP. We run it with its three perturbation types on the same twins and give it access to a training split wherever one exists, as in FinEval. Its reported real-world result is detection of likely contamination in Baichuan-13B and Disc-Fin-13B on FinEval, confirmed afterwards by a 21-question train–test overlap. No reported AUROC for CAP was found on Lacuna, so beating it means matching that FinEval verdict while achieving higher AUROC on our injected-contamination splits, where CAP and our method see identical twins and inference budgets.

The strongest gray-box competitors are PaCoST and Min-K% Prob, run on the open panel members with the same rephrasings. PaCoST reports dataset-level significance at $p<0.05$, and Min-K% failed completely on chain-of-thought-augmented MMLU leakage in the Oracle study. Beating them means a significant benchmark-level detection where they do not, under that augmentation. We also report AECA and LOS-Net where weights permit, noting that LOS-Net's 0.95 AUC on BookMIA was measured on a different task (membership of long book passages) and is not directly comparable.

Two ablations isolate our ingredients. One removes the panel, leaving the gap corrected only for fluency. The other removes the fluency covariate, leaving the panel correction only. If either ablation matches the full method, that ingredient is unnecessary.

Why not just…?

Why not just use the single-model twin gap, as PaCoST and CAP do? Because a nonzero gap is expected even for clean models whenever twins are slightly harder or less fluent. DyVal shows that GPT-4 loses 13.6% accuracy on reachability tasks merely when description order is flipped, so surface changes alone move accuracy substantially. Our anchors test this directly: if clean anchors show near-zero raw gaps, the panel is unnecessary and we would say so. We expect, but have not shown, that they will not.

Why not just compare models on the original items and flag those that beat their capability-predicted accuracy? This is pure cross-model differential testing, an item-response-style residual, and it is a real threat because it is cheap. Its weakness is that it cannot separate "item is in model $m$'s data" from "model $m$ has a specialty," for example a finance-tuned model on FinEval. Twins remove that confound because a specialty transfers to twins while memorization does not. We include this residual as a baseline. If it equals our method on the controlled fine-tunes, the twin construction is not earning its cost.

Why not just have humans write the twins and skip machine-text detection? Human twins would remove the fluency confound at its source, so on a gold subset of a few hundred items we will commission them. If LRR-matched machine twins and human twins give the same $\kappa$ rankings, the detector covariate is validated as a cheap substitute. If they diverge, the fluency model is inadequate, and that result is itself informative about the Fluency Trap.

Why not just use dynamic benchmarks like DyVal and stop auditing static ones? DyVal works only where problems can be specified formally. Static natural-language benchmarks such as MMLU remain in use and remain the place where contamination matters most. We use DyVal only as the cleanest twin source and as a sanity check.

Decisive Experiments

The first test uses controlled fine-tunes on MMLU and GSM8K, with half the items leaked and the other half clean, in verbatim and chain-of-thought-augmented variants. We measure item-level AUROC and benchmark-level detection power against all baselines. Success requires a clear benchmark-level signal on the augmented variant, where Min-K% is known to fail.

The second test uses WMDP, released after the cutoff of most panel models. Clean models must show $\kappa$ near zero, and a deliberate WMDP fine-tune, which PaCoST also used, must be flagged. Together these give the false-positive rate and the power.

The third test uses FinEval, where we must recover the confirmed Baichuan-13B case without using a training split.

The fourth test is a shared-contamination stress case: we fine-tune most panel members on the same leaked half. This quantifies how badly the collusion failure degrades detection and how much the public-data anchors restore it.

The method fails if the panel-free ablation performs as well, or if the stress case shows that realistic shared leakage erases the signal even with anchors present.