Contamination as Displacement in a Stable Map of Models and Questions

When a language model has seen a benchmark's test questions during training, its score on those questions is inflated. The best current evidence of this is behavioral. A contaminated model does much worse on rewritten or structurally perturbed versions of the same questions than its overall ability predicts. Today that evidence is compressed into one number per model and benchmark. We propose to keep the full pattern instead: which models get which questions right, under the original and under each perturbed variant. We treat each variant as one time slice of an evolving network that links models to questions, and we embed all slices jointly using a dynamic network embedding method with proven stability, so that positions are directly comparable across slices. In that shared map, an honest model moves the way the rest of the population moves when questions get harder. A contaminated model, or a leaked question, moves differently. That anomalous movement is the contamination signal.

This matters because the most widely studied detectors are failing. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a brief reinforcement learning phase after contaminated fine-tuning pushes likelihood-based detectors toward chance. The same collapse follows chain-of-thought contamination of reasoning models. In both cases the score inflation survives. Our detector never reads token probabilities, so this cloaking mechanism does not act on it directly. It also gives question-level and model-level evidence with a null distribution, rather than a single gap.

Core Claim

Consider the correctness patterns of a large population of models on an item family, meaning an original question plus its structurally perturbed variants. Embed these patterns jointly by unfolded spectral embedding. Contamination then appears as a model's or an item's displacement between the original slice and the perturbed slices that departs from the population-wide trend. We expect this displacement test to detect sample-level contamination more reliably than a scalar performance gap, with less dependence on a hand-picked panel of trusted reference models. We also expect it to remain informative after reinforcement learning post-training, where likelihood detectors fail.

The key technical ingredient is the stability property of unfolded embeddings. Spectral embedding for dynamic networks with stability guarantees (UASE) concatenates snapshot adjacency matrices side by side and takes one singular value decomposition. It proves two things. First, a node whose connection behavior is unchanged across snapshots keeps the same coordinates. Second, nodes that behave identically within a snapshot share coordinates. Both hold up to Gaussian noise with a known asymptotic form. Embedding each slice separately cannot give this. Such embeddings are only defined up to rotation, so movement between slices mixes real change with arbitrary realignment, the "latent space drift" noted by Simou et al..

Research Program

Let $A^{(t)} \in \{0,1\}^{n \times m}$ record whether model $j$ (of $m$ models) answers variant $t$ of item family $i$ (of $n$ families) correctly. Slice $t=1$ is the original benchmark. Slices $t \ge 2$ are perturbations. Model-level analysis uses the unfolding

$$\mathbf{A} = [A^{(1)} \mid A^{(2)} \mid \dots \mid A^{(T)}] \in \mathbb{R}^{n \times mT},$$

and its rank-$d$ singular value decomposition gives a per-slice embedding $\hat Y^{(t)}_j \in \mathbb{R}^d$ for each model. Item-level analysis unfolds the transposed matrices, which gives per-slice embeddings $\hat X^{(t)}_i$ for each item family.

Positions will shift for every model when variants are harder. So the statistic is not raw displacement. It is the residual after a robust regression of $\hat Y^{(t)}$ on $\hat Y^{(1)}$ fitted across the whole population:

$$r_j = \big(\hat Y^{(t)}_j - \hat f(\hat Y^{(1)}_j)\big)^\top \hat\Sigma_j^{-1} \big(\hat Y^{(t)}_j - \hat f(\hat Y^{(1)}_j)\big).$$

Here $\hat f$ is the robust population trend. The matrix $\hat\Sigma_j$ is the plug-in estimate of the node's covariance from the UASE central limit theorem. If that theorem holds approximately, $r_j$ is approximately chi-squared under "no contamination," which gives calibrated p-values. The same construction on the item side flags leaked questions.

This is a multivariate, population-referenced generalization of the difficulty-correction idea in ConStat. ConStat fits a scalar curve from a trusted reference panel. We fit a $d$-dimensional trend robustly across all models, so a minority of contaminated models cannot drag the reference. This directly targets ConStat's stated weakness of relying on clean reference models.

The perturbed slices come from structural generators rather than paraphrase alone. We use depth, width and numerical perturbations of reasoning graphs from DARG, with code-executed labels. We use DyVal-style controlled graph generation from DyVal for families built from scratch. We keep ConStat-style rephrasings as a separate slice.

Every slice has strong degree heterogeneity: strong models connect to most items, and easy items connect to most models. This is exactly the regime where Unfolded Laplacian Spectral Embedding reports unfolded adjacency embedding degrading (ARI ~0.54) while normalized unfolded Laplacians recover structure (ARI/NMI ~0.9–1.0). We therefore run both UASE and ULSE as a planned comparison, not an afterthought.

The contribution is thus one concrete construction and three analyses. The construction is the unfolded model–item network over perturbation slices. The analyses are, first, whether the displacement test detects planted contamination; second, whether it survives reinforcement learning cloaking; and third, which ingredients (joint unfolding, degree normalization, structural perturbations) carry the signal.

Baselines

The naive reference point is the raw paired accuracy drop. For each model, subtract its perturbed-slice accuracy from its original-slice accuracy and rank models by this drop. For items, rank each family by the fraction of models that solve the original but fail the variants. This is what DyVal and DARG implicitly report. Any structured method that fails to beat it is not worth its complexity. It is computed on exactly the same response matrices.

The strongest existing performance-based method is ConStat. It reports detecting 98.4% of sample-specific contamination in controlled experiments, against at most 71% for the baselines it compared. In the wild it reports gaps of δ≈8.3% for Mistral-7b-v0.1 on GSM8K and δ≈6.6% for Llama-3-70b on ARC. We run ConStat on the same perturbed slices we use, with its reference panel drawn from the same model population. We also run a variant with a deliberately contaminated fraction of the reference panel, since our claim is about robustness to exactly that. Beating ConStat means higher detection power at a matched false-positive rate when the reference panel is partly contaminated, while not losing to it when the panel is clean.

The strongest likelihood-based detector is LiRA, as evaluated in the Fragility paper. There it reaches 89.1% AUROC before reinforcement learning and falls sharply after one clean GRPO epoch; reference-free detectors fall to ~50%. We reproduce its Stage I protocol and score LiRA and Min-K% on the same planted models. Beating it means an AUROC clearly above its post-reinforcement-learning value on the same checkpoints.

The strongest simple psychometric competitor is multidimensional item response theory (IRT), the standard statistical model of test answers that gives each model a latent ability and each question a latent difficulty. We fit it jointly across slices, with item-family parameters linked across variants and a per-slice ability shift. Residuals are scored the same way as ours. No reported result found on Lacuna for IRT-based contamination detection. It receives the same data, the same latent dimension search and a comparable tuning budget.

Two ablations isolate our key ingredients. The first embeds each slice independently and aligns slices by orthogonal Procrustes. This tests whether joint unfolding with its stability guarantee matters or whether post-hoc alignment suffices. The second swaps ULSE for UASE, which tests whether degree normalization matters under the heavy accuracy heterogeneity of real model populations.

Why not just…?

Why not just use ConStat's scalar gap? ConStat is a real competitor and may match us when a clean reference panel exists and contamination is model-wide. Our argument concerns two regimes it was not designed for. The first is sample-level leakage that is diluted in an aggregate score. The second is reference panels that are themselves contaminated, which ConStat's authors flag as a limitation. The contaminated-reference experiment below settles this directly. If ConStat holds up there, our contribution shrinks to item-level localization.

Why not just fit a multidimensional IRT model? This is the most serious threat. IRT with linked items is conceptually close to what we do, and spectral embedding of a binary response matrix can be seen as a fast, non-parametric cousin of it. The possible advantages of the spectral route are three: no parametric link function, a direct asymptotic test statistic, and known behavior under degree heterogeneity via ULSE. None of these is guaranteed to matter in practice. We run IRT as a full baseline. If it matches our detection power, we will report the spectral method as a scalable approximation with calibrated tests, not as a better detector.

Why not just improve likelihood-based detectors? The Fragility paper argues that clipped policy-gradient updates equalize loss statistics between seen and unseen questions. It also finds that chain-of-thought contamination produces generalization rather than verbatim recall. Both findings attack signals that live in token probabilities. Correctness patterns are a different signal, and the same paper shows the performance inflation persists (8.8% after Stage I, 11.7% in Stage II). There is an honest caveat: that generalization to similar unseen items may also shrink the original-versus-perturbed gap we rely on. The Stage II experiment measures exactly how much.

Why not just paraphrase questions? DARG finds that numerical changes mostly produce calculation errors, while depth increases cause logical collapse; Claude-3-Opus lost over 50% accuracy with four extra steps. Paraphrase slices may therefore move honest and contaminated models similarly. We keep paraphrase as its own slice and report detection power per slice type, which shows whether structural perturbation is needed.

Proposed Experiments

The core experiment is a controlled planting study on GSM8K and on DARG-perturbed GSM8K and BBH (Navigate, Dyck) families. We take a population of open-weight models and fine-tune a subset on randomly chosen subsets of original test items at several leak fractions. This gives ground truth at both the model level and the item level. Following the Fragility Stage I protocol on Qwen2.5-7B-Instruct, a further subset then receives one epoch of clean GRPO. For the Stage II protocol, DeepSeek-R1-Distill-Llama-8B is contaminated with chain-of-thought traces on AIME, AMC and OlympiadBench items, whose perturbed variants we generate where the reasoning graph is extractable.

We report model-level and item-level AUROC and detection power at 5% false-positive rate for every baseline and ablation. We also check calibration of the chi-squared p-values on clean models.

Three decisive checks follow. The first is whether our AUROC on post-GRPO and Stage II models stays well above the ~50–58% the likelihood detectors reach on the same checkpoints. If not, behavioral displacement is also cloaked, and the central motivation fails. The second is whether performance holds as the fraction of contaminated models in the population rises. We expect the robust trend to break down around the regression's breakdown point, and ConStat to degrade as soon as its panel is contaminated. The third is whether ULSE beats UASE and joint unfolding beats Procrustes alignment. If neither does, the dynamic-embedding machinery is not earning its place, and IRT or raw residuals should be preferred.

A final out-of-sample check applies the method to public models on GSM8K and ARC. It tests whether our flags agree in direction with ConStat's published gaps for Mistral-7b-v0.1 and Llama-3-70b, and which specific items drive them.