When a language model has trained on a multiple-choice benchmark, it has seen each question with its answer options in one particular printed order. A model that never saw the item has no reason to prefer that order over any reshuffling of the same options. This gives a test for contamination. For each benchmark item, score the original layout and a set of option-shuffled copies under the model, and ask how often the original ranks first. If the benchmark is clean, the original's rank among its shuffles is uniformly random. The resulting p-value is exact, needs no clean reference set, and does not care whether the training pipeline shuffled the order of whole items.
The proposal builds on that test with two additions from the statistics of high-dimensional testing. The first is the "learn-then-test" recipe. Fit a scoring function that squashes a long vector of per-token model probabilities into one number, using one half of the items. Then run a simple exact rank test on the other half. The labels for that fitting step come for free, because "original layout" versus "shuffled layout" is known for every item. The second addition is a way of combining items that targets sparse contamination, where only a few percent of items leaked. A plain average would wash that signal out.
Core Claim
The only existing contamination test with exact error control on black-box models is the sharded rank comparison of Proving Test Set Contamination in Black-Box Language Models. It compares the likelihood of the benchmark's canonical item order against permuted orders, shard by shard, then runs a t-test. It detects datasets duplicated only 2 to 4 times in 1,000-example test sets and reaches $p < 10^{-38}$ at ten or more duplications. The authors acknowledge that the signal vanishes if items were shuffled or ingested one page at a time, which is how benchmark items usually reach web corpora.
Our claim is that exchangeability can be moved from the level of the dataset to the level of the single item. The within-item option order survives every between-item shuffle, because each scraped copy of a question carries its options verbatim. Three properties follow:
- the per-item test is exact for any scoring function, including one learned from data, as long as that function was fitted on other items;
- the scoring function can be learned without any labeled clean or contaminated examples;
- items can be aggregated with a statistic built for the sparse-mixture regime that When Is Benchmark Contamination Detectable? identifies as the realistic one.
That paper models the audited benchmark as $Q_\alpha=(1-\alpha)P_0+\alpha P_1$, a mixture in which a fraction $\alpha$ of items was seen. It proves detection power is limited by roughly $\alpha\rho\sqrt{m}$, where $\rho$ is how strongly a seen item changes model behavior and $m$ is the number of items. It also requires matched clean controls, which outside auditors of commercial models rarely have. Our test needs none, because each item's shuffled copies serve as its own controls.
Research Program
The per-item test. For item $i$, let $g_0$ be the canonical layout. Let $g_1,\dots,g_{K}$ be layouts drawn from a permutation group $G_i$ of that item's options. Let $s$ be any scoring function mapping the model's per-token log-probabilities on a layout to a scalar. The per-item p-value is
$$p_i=\frac{1+\#\{k\ge 1: s(g_k)\ge s(g_0)\}}{K+1},$$
with ties broken at random. If the model's distribution over the layouts in $G_i$ is exchangeable, $p_i$ is uniform on $\{1/(K+1),\dots,1\}$. This holds for any fixed $s$. It is the same fact that makes the sharded test exact, applied inside an item.
We use two permutation groups:
- The full group permutes all options; with 4 options this gives 24 layouts, so the smallest possible $p_i$ is 1/24.
- The distractor-only group keeps the correct answer in its printed slot and permutes only the wrong options; this gives 6 layouts.
The distractor-only group removes a confound. A capable but clean model might, through learned position preferences such as those documented in ABCD: All Biases Come Disguised, assign different likelihoods depending on where the correct answer sits.
The learned projection. The baseline choice of $s$ is the total log-likelihood of the layout. Most of that total comes from tokens whose probability does not depend on option order. The memorization signal is likely concentrated near option boundaries and in the first tokens of each option.
Following AUC optimization and the two-sample problem, we treat canonical layouts and shuffled layouts as two samples. That paper learns a scoring function on one half of the data by maximizing a ranking score, and runs a Mann-Whitney-Wilcoxon rank test on the other half. In its Gaussian experiments this reached 85% power against 54–58% for kernel two-sample tests (maximum mean discrepancy). We fit a small pairwise ranking model on half of the items, trained to rank $g_0$ above its shuffles. Its inputs are per-token features:
- raw log-probability;
- the vocabulary-normalized z-score from Min-K%++, which standardizes each token's log-probability by the mean and spread across the vocabulary at that position;
- the token's position relative to option boundaries.
The fitted model is then frozen and applied to the held-out half. We cross-fit by swapping the halves, and combine the two halves' aggregate p-values as twice their minimum. That combination stays valid under arbitrary dependence and limits the cost of splitting the sample, the main power penalty both AUC optimization and A Permutation-Free Kernel Independence Test flag for split-sample tests. Our setting differs from Clémençon et al. in one respect: there, the two samples are given, whereas here they are manufactured from the null's own symmetry.
Aggregation under sparsity. With $\alpha$ small, most $p_i$ are uniform. A seen item tends to put its canonical layout at rank one. We therefore aggregate with an exact binomial count of top-ranked items:
$$T=\#\{i: p_i = 1/(K+1)\}, \qquad T\sim \mathrm{Binomial}\!\left(m,\tfrac{1}{K+1}\right)\ \text{under the null}.$$
This works because, with independent random tie-breaking, items are mutually independent under the null. We compare $T$ against Fisher's combination and the mean p-value. We also compare it against a tail-weighted rank statistic in the spirit of the one-tailed weighted-AUC test of Test for non-negligible adverse shifts, which places quadratic weight on the extreme tail of an outlier score so that bulk shifts do not dominate. The roles of the cited work are as follows:
- Kamulete's tail-weighted test supplies the aggregation idea.
- Shihab et al. supply the sparse-mixture model and the $\alpha\rho\sqrt{m}$ ceiling we measure against.
- Their finding that textbook Gaussian sample-size formulas failed in 9 of 9 small-sample cases constrains how we plan audits. We use exact binomial power calculations rather than normal approximations.
Validity analysis. A learned $s$ will exploit any way in which canonical option orders are not arbitrary, for example options listed in ascending numeric order, alphabetical order, or "all of the above" placed last. This would produce false contamination findings. We therefore treat null validity as a measured quantity. We filter items whose options have a fixed logical order, or restrict $G_i$ to orders consistent with that structure. We then report empirical false-positive rates on items the model provably never saw.
Baselines
Naive reference points. The do-nothing reference is the fixed total-log-likelihood statistic inside our own permutation test, aggregated by mean p-value. It isolates what learning and tail-aware aggregation add. A second naive reference is temporal accuracy decay, run on the same models, which Test of Time showed can be erased by rephrasing.
Strongest existing test. The strongest competitor with exact error control is the sharded rank comparison test of Proving Test Set Contamination. Its reported numbers are reliable detection at 2–4 duplications on 1,000-example sets, $p<10^{-38}$ at ten or more duplications, $p=0.001$ for Mistral-7B on AI2-ARC, and $p\approx0.01$ for MMLU on LLaMA2 and Mistral. We run it with the authors' shard settings on exactly the injected models and item sets we test, at an equal number of forward passes. Beating it means two things:
- comparable power when the canonical item order was preserved in training;
- high power where it reverts to chance because items were shuffled. We found no reported result on Lacuna for the sharded test under shuffled ingestion, so that number will be measured here.
Strongest detection scores. The strongest score-based detector is Min-K%++, which reports 6.2 to 10.5 AUROC points over Min-K% on WikiMIA. We run it two ways. As a stand-alone detector, it is thresholded against a post-cutoff control set when one exists. As a fixed $s$ inside our permutation test, it isolates whether the learned projection beats the best hand-designed one. We run Polarized Augment Calibration from Data Contamination Calibration for Black-box LLMs, a polarized max-minus-min token-confidence gap over word-swapped variants, the same two ways. It reports 4.5 and 5.9 AUC points of improvement on WikiMIA and StackMIA.
Fairness and the decisive ablation. All detectors get identical items, identical forward-pass budgets, and the same held-out tuning split for any hyperparameter, including Min-K%++'s $k$. The learned-versus-fixed comparison is the ablation that tests the method's key ingredient. If Min-K%++ or total log-likelihood inside the permutation test matches the learned projection, we will report that the within-item test with sparse aggregation is the contribution and the learning step is unnecessary.
Why not just…?
Why not just run the sharded order test? It is exact and already deployed, and on datasets ingested in canonical order it may beat us, since it pools ordering information across many items. Its own authors note that it collapses once item order is destroyed, and real leakage through scraped pages rarely preserves it. Our injection study contains both regimes. A result where the sharded test wins on canonical-order injection and fails on shuffled injection, while ours holds in both, settles the question. If ours fails in both, the idea is wrong.
Why not just threshold Min-K%++ or PAC scores? These give rankings, not p-values. Thresholding them requires a clean reference distribution, which is exactly what When Is Benchmark Contamination Detectable? shows is usually missing, and why it finds "no evidence" audits often have near-zero power. Their per-item signal may still be excellent. That is why we also plug them into our permutation test as fixed scores, where they inherit exact validity. If they match the learned score there, we adopt them.
Why not just use the fixed total log-likelihood and skip learning? This is the most serious threat. Rank-among-shuffles of total likelihood is simple and may capture most of the signal for heavily duplicated items. The learned projection should help most at low duplication counts, where memorization shows up only in a few boundary tokens. The duplication sweep below measures exactly this gap. If the gap is under a few points of power everywhere, we drop the learning step from the recommended method.
Why not just use influence functions? Test of Time used them on OLMo2-7B-Instruct. Even there they were limited to 40 papers by compute, and their hit rate fell from 77.5% to 17.5% under rephrasing. They also need weights and training data, which black-box audits lack.
Proposed Experiments
Controlled injection. Following the injection designs of both Proving Test Set Contamination and When Is Benchmark Contamination Detectable?, we continue pretraining a roughly 1.4B-parameter model on a background corpus. We use the Pile or FineWeb. Into it we inject subsets of MMLU, AI2-ARC and HellaSwag, crossing three factors:
- contamination fraction $\alpha$ from 1% to 100% of the audited set;
- duplication counts of 1, 2, 4 and 10;
- three ingestion modes: canonical item order, shuffled item order with options kept verbatim, and shuffled items with options reshuffled at each occurrence.
The third ingestion mode is a built-in negative control. The model has seen the content, but no canonical option order exists to memorize, so a valid test should not fire. We repeat over five training seeds. Primary metrics are:
- power at a 5% false-positive level, as a function of $\alpha$ and duplication;
- empirical false-positive rate on never-injected items;
- the ratio of observed power to the $\alpha\rho\sqrt{m}$ ceiling.
Null validity on real models. We apply the full pipeline, including the learned score, to multiple-choice items written after each model's training cutoff and to freshly generated items. On these the false-positive rate must sit at or below nominal. We do this separately for the full and distractor-only groups, before and after structural filtering. Any inflation here would invalidate downstream claims, so this check gates everything else.
Real audits. We audit LLaMA2-7B, Mistral-7B, Pythia-1.4B and GPT-2 XL on MMLU and AI2-ARC, the same models where the sharded test reported its MMLU and ARC p-values, and report agreement and disagreement with those results. We then extend to open models whose training data can be searched, so that positive findings can be checked against actual occurrences of the items.
Scope. The method needs sequence log-probabilities. For commercial interfaces that hide them, the logit_bias probing of Data Contamination Calibration is too costly at 24 layouts per item, so the first study is limited to open-weight and log-prob-exposing models. The test also targets verbatim item ingestion only. Paraphrased contamination, which Shihab et al. rank below verbatim exposure in detectability, is outside what option-order symmetry can see. We will state this rather than claim it.
References
- Proving Test Set Contamination in Black-Box Language Models
- ABCD: All Biases Come Disguised
- AUC optimization and the two-sample problem
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- A Permutation-Free Kernel Independence Test
- Test for non-negligible adverse shifts
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Data Contamination Calibration for Black-box LLMs