Geometric Fingerprints of Benchmark Contamination Inside Reasoning Traces

Contamination detectors for language models usually ask a single scalar question: does the model find this test item suspiciously easy, meaning low loss, high confidence, or near-identical outputs? That question stops working in the setting that now matters most. After a contaminated model goes through reinforcement learning, or when it was trained on worked solutions rather than bare answers, the gap in token probabilities between seen and unseen items largely disappears, and detectors fall to near chance. We propose to look at a richer signal than confidence. As the model works through a benchmark item, its hidden states trace out a cloud of points, and the shape of that cloud can be summarized by its covariance matrix. The hypothesis is that memorized items produce a cloud that is characteristically narrow and distorted, because the model retrieves a stored trajectory instead of computing one, and that this change in shape survives even when output probabilities have been flattened.

Comparing covariance matrices faithfully requires geometry. A covariance built from a few hundred tokens in a 4096-dimensional hidden space is singular, so it cannot be compared with distances designed for positive definite matrices. Flattening it and running standard PCA distorts volumes and can yield matrices with negative eigenvalues. We borrow a tool from manifold-preserving dimensionality reduction to handle this: an orthonormal bilinear projection, $Y = W^\top X W$, which compresses each large covariance $X$ into a small matrix $Y$ that is guaranteed to remain a valid covariance. We then compare these small matrices using the distance appropriate to that space. The resulting detector needs no training data, no clean reference models, and no sampling of many generations. It is built for one question: does trace geometry expose contamination in the regimes where On The Fragility of Benchmark Contamination Detection in Reasoning Models showed that loss-based detectors fail?

Core Claim

For each benchmark item $i$, let $\Sigma_i$ be the covariance of layer-$\ell$ hidden states over the tokens of the model's own reasoning trace. Let $W \in \mathbb{R}^{D \times d}$ be an orthonormal frame ($W^\top W = I_d$, with $D$ the hidden width and $d \approx 32$–$64$), fitted once per model on clean text published after the model's training cutoff. We form

$$C_i = W^\top \Sigma_i W + \epsilon I_d ,$$

where $\epsilon$ is a small eigenvalue floor, as in the rectification layer of SPDNet. The contamination score is

$$s_i = \big\| \log\!\big(\bar C^{-1/2} C_i \bar C^{-1/2}\big) \big\|_F ,$$

the affine-invariant geodesic distance from $C_i$ to $\bar C$. Here $\bar C$ is the Riemannian mean of the $C_j$ over difficulty-matched clean items, $\log$ is the matrix logarithm, and $\|\cdot\|_F$ is the Frobenius norm.

We claim three things. First, $s_i$ separates members from non-members of a contaminated benchmark. Second, its AUROC degrades much less than the AUROC of LiRA or raw loss after clean GRPO, a reinforcement-learning method for reasoning models. Third, the full geodesic distance carries information beyond two scalar summaries: the volume of the cloud (its log-determinant) and its intrinsic dimension.

The third claim is where the manifold machinery either earns its place or fails. The proposal is built so that this question is settled directly.

Why this is plausible, and what it is not

Three strands of prior work motivate the hypothesis. Memorization in Language Models through the Lens of Intrinsic Dimension found that text whose BERT embeddings have higher intrinsic dimension is memorized less. It measured this with TwoNN, a nearest-neighbor dimension estimator, on static embeddings of general web text. We instead measure the geometry of the audited model's own hidden states during generation, on benchmark items under evasive contamination. Generalization or Memorization showed that contamination makes output distributions "peak," meaning repeated samples collapse toward the same text. We look for the internal counterpart of that collapse, and our method needs one generation rather than many samples. Think or Remember? found that memorization and generalization activate different deep-layer neurons, but it compares per-neuron means on synthetic tasks. We use second-order structure on real benchmarks.

The closest connecting work is Parameter Manifold Purification. It projects the corrupted weights of vision models back onto a learned manifold of clean weights, using an autoencoder and diffusion. It operates on weights rather than activations, it repairs models rather than auditing them, and it does not touch language models or benchmark leakage. The method side is supplied by SPDNet, which contributes the bilinear map and the eigenvalue floor. Geometry-Aware visualization of high dimensional SPD matrices and Chevallier et al. supply the evidence that Euclidean averaging of covariances inflates volumes (the "swelling effect") and distorts neighborhoods.

The Fragility paper is the main constraint on this proposal. It shows that CoT-contaminated reasoning models generalize to rephrased variants of the leaked items. That kills any detector built on brittleness, including a naive version of ours that pairs each original item with a paraphrase. For this reason our primary statistic compares against a difficulty-matched clean-item null, not against the item's own paraphrase. Paraphrase pairing is kept only as a secondary diagnostic.

Research Program

We first build the extraction pipeline. For open-weight models we greedy-decode each item, record hidden states at every layer, form per-item token covariances, and project them with a frame $W$ taken from the top-$d$ eigenvectors of the pooled covariance on post-cutoff clean text. Because $d$ is small, eigendecompositions cost almost nothing. This sidesteps the cubic-cost problem that limits Riemannian methods at foundation-model scale.

We then test the hypothesis layer by layer under controlled contamination. Last, we test a learned variant, in which $W$ is trained by Riemannian gradient descent on the orthonormality constraint, as in the SPDNet bilinear layer. It is trained on shadow contamination: the auditor fine-tunes a copy of the model on a disjoint set of held-out items. This tells us whether learning the projection helps beyond the unsupervised frame. It also exposes whether a learned detector overfits to the shadow setup.

Baselines

The do-nothing reference is chance, 50% AUROC. We also report the benchmark-accuracy gap between contaminated and clean copies, which tells us how much inflation there was to detect in the first place.

The strongest white-box competitor is LiRA, as run in On The Fragility of Benchmark Contamination Detection. That paper reports 89.1% AUROC for LiRA on SFT-contaminated Qwen2.5-7B-Instruct before RL. The paper reports that LiRA loses about 14 points after one epoch of clean GRPO, the raw loss detector loses about 16, and all detectors average 50–58% AUROC on the CoT-contaminated DeepSeek-R1-Distill-Llama-8B. We rerun LiRA and loss on exactly our contaminated checkpoints and member/non-member splits, giving them the same number of reference or shadow models we give our learned variant. Beating it means matching LiRA before RL within noise, and then losing substantially less AUROC than LiRA after GRPO. In the CoT setting, a meaningful win is a score clearly above the 58% ceiling reported for existing methods.

The strongest confidence-based competitor is PaCoST, which applies a paired t-test to the probability the model assigns to "Yes" when asked whether an answer is correct, comparing original and rephrased questions. We run it with the same paraphrases we use for our secondary diagnostic. On surface-perturbation methods, we run CDD with the sampling budget from its paper. That paper reports 21.8–30.2% relative improvement over earlier baselines on DETCON, but no absolute AUROC for our reasoning setting; no reported result found on Lacuna for CDD under RL cloaking. ConStat reports 98.4% detection in controlled experiments. However, it estimates score inflation for a whole benchmark against clean reference models, so we compare against it only at benchmark level.

Four ablations isolate the geometric ingredient, all using the same $W$ and layer. The first, scalar volume, uses the log-determinant of $C_i$ alone. The second, scalar dimension, runs TwoNN on the raw hidden states of the trace. The third is Euclidean: flattened covariances projected by PCA and compared with Frobenius distance, with no positive-definite guarantee. The fourth, log-Euclidean, uses shrinkage (Ledoit–Wolf) covariance in the full space followed by the matrix logarithm, which keeps positive definiteness but skips the low-dimensional projection.

Why not just…?

Why not just train a linear probe on mean-pooled hidden states? A probe needs labeled members and non-members of the target model, which a real auditor lacks. With shadow contamination it becomes a fair comparison, so we include it alongside our learned variant. If the probe matches our training-free score after RL, the geometric story is unnecessary for supervised audits. We would report that plainly and restrict the claim to the label-free setting.

Why not just measure intrinsic dimension or log-determinant, following Arnold? This is the most serious threat. Contamination might simply shrink the volume of the trace cloud, and a scalar would then capture everything. The scalar-volume and scalar-dimension ablations test this directly. The geodesic score has to beat both at matched layers, especially after GRPO, for the manifold machinery to be justified. If it does not, we will reframe the contribution as "scalar trace geometry survives RL cloaking," which would still be a useful result.

Why not just use perplexity-style detectors, or move to fresh private benchmarks? The Fragility paper already shows the first failing in exactly our regime. That paper found clean GRPO removes most of the log-probability gap, and CoT contamination pushes ten detectors to 50–58%. Fresh benchmarks help in the future but cannot audit the scores already reported on MMLU, GSM8K, or ARC. They are also costly to build at expert level, as the expert-level academic benchmark itself illustrates.

Proposed Experiments

Stage one reproduces the Fragility protocol on Qwen2.5-7B-Instruct. We SFT-contaminate half of each of AIME, AMC, GPQA Diamond (GPQA), OlympiadBench, and Minerva Math, then apply zero, one, and three epochs of clean GRPO. We report member/non-member AUROC and TPR at 1% FPR, per layer and per epoch. The decisive check is the AUROC drop from zero to one epoch of GRPO for our score versus LiRA's roughly 14-point drop.

Stage two applies CoT contamination to DeepSeek-R1-Distill-Llama-8B. The decisive check is whether any version of the score clears 58% AUROC. We also check whether the paraphrase-paired diagnostic fails as predicted, which would confirm that the clean-null design was needed.

Stage three uses Evasive Augmentation Learning, from Evading Data Contamination Detection, on MMLU (Hendrycks et al.), ARC (Clark et al.), and GSM8K (Cobbe et al.). The reference point is the roughly 1% TPR at 1% FPR that paper reports for sample-level detectors.

As a sanity control, two uncontaminated copies of each model, trained with the same SFT and RL budget on unrelated data, must score near 50%. This rules out the possibility that the score merely tracks fine-tuning or item difficulty. The overall result is falsified if, after GRPO, the geodesic score's AUROC is no better than the log-determinant ablation. It is also falsified if the score falls to the same level as LiRA.