When a language model scores well on a math benchmark, we want to know whether it solved the problems or had already seen them during training. The usual detectors look at the model's output probabilities and ask whether a test question looks "too familiar." These detectors stop working once the model is further trained with reinforcement learning, because that training reshapes output probabilities for seen and unseen questions alike. We propose a different check. Run each benchmark question once through the model before and after post-training, collect the internal activation vector for every token, and summarize how the two sets of activations differ from a fixed statistical description of ordinary text. The summary is a Fisher vector: a fixed-length record of how a set of vectors shifts and tightens relative to a Gaussian mixture fitted on reference data. Our central hypothesis is that memorized questions show up as a specific signature: the token activations for those questions bunch more tightly around the mixture's clusters after post-training than activations for comparable unseen questions do.
The opportunity matters because the most decision-relevant evaluations, such as AIME-style math and science reasoning, are now run on reinforcement-learning-tuned models, and this is where output-level detection has collapsed toward chance. The method needs one forward pass per model per question and no paraphrasing, question blanking, or image synthesis. It produces a score for each question rather than one verdict for the whole benchmark, and the score has an interpretable second-order component that can be checked against a mechanistic prediction.
Core Claim
Benchmark contamination that survives post-training lives in the spread of token-level hidden states, not only in their average location. Take the frozen base model, collect token hidden states on a generic reference corpus, reduce them with principal component analysis (PCA), and fit a diagonal Gaussian mixture model to the reduced states. In the Fisher vector literature this fitted mixture is called the universal background model. Using this one mixture, encode the same question's token states from the base model and from the post-trained model, and take the difference. Fixing the mixture and differencing the same text through both models cancels what the question is about and how it is written, and leaves what training did to it. Write $\gamma_t(k)$ for the posterior weight of mixture component $k$ for token state $x_t$, with $t = 1, \dots, T$ indexing the question's tokens. Write $\mu_k$ and $\sigma_k$ for the component's mean and per-dimension standard deviation, and $w_k$ for its mixture weight. The variance part of the Fisher vector is then
$$G^{\sigma}_k(X) = \frac{1}{T\sqrt{2w_k}} \sum_{t=1}^{T} \gamma_t(k)\left[\frac{(x_t-\mu_k)^2}{\sigma_k^2} - 1\right],$$
and the audit statistic for a question is
$$\Delta(q) = \mathrm{FV}_{\text{post}}(q) - \mathrm{FV}_{\text{base}}(q),$$
which is then scored against the null distribution of $\Delta$ on questions known to be clean. The prediction that makes this falsifiable is directional. Contaminated questions should have systematically negative $\Delta$ in the variance block $G^{\sigma}$, meaning their activations tighten, at middle-to-late layers. We expect this because LaRA reports that contaminated items' internal states are unusually "stiff" and collapse onto low-dimensional directions. A positive result is a variance-block detector that beats both a means-only Fisher vector and mean-pooled activation differences at low false-positive rates.
Relation to Prior Work
Two papers motivate the problem. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates reasoning models through supervised fine-tuning followed by GRPO, a reinforcement-learning method. It shows that loss, Min-K% and LiRA detectors degrade sharply after even one GRPO epoch. When the contamination enters through chain-of-thought training data (worked solutions written out step by step), ten output-level detectors average roughly 50–58% AUROC. This rules out any design that relies only on output log-probabilities, and it is why we work inside the network. Soft Contamination Means Benchmarks Test Shallow Generalization shows that semantic near-duplicates evade n-gram filters yet inflate scores by 15–20%. Its tool is cosine similarity against an open training corpus, which says nothing about a closed model. Our detector inspects the model instead of the corpus, and one experiment below reuses its soft-contamination regime.
The closest prior work is LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training. LaRA also compares pre- and post-trained hidden states. It computes three hand-designed geometric scores and calibrates them against a clean reference using median absolute deviation. Its scores come from controlled perturbations of each question: paraphrased neighbors and versions with key facts blanked out. LaRA and our proposal share the premise that the signal is internal. We differ in three ways. We use no perturbations. We replace hand-designed distances with a probabilistic model of the token-state distribution and its gradient statistics. And we isolate a specific second-order term rather than combining several heuristic scores. If our variance-block result does not beat LaRA, the probabilistic machinery has not earned its place.
The method comes from the Fisher vector line. XRCE's Participation to ImageCLEF 2008 supplies the encoding, which records first- and second-order deviations from a mixture. Fisher vectors with cascaded normalization for paralinguistic analysis supplies the power-plus-L2 normalization that makes variable-length sequences comparable. Aggregation Schemes for Single-Vector WSI Representation Learning in Digital Pathology and Species-Agnostic Patterned Animal Re-identification by Aggregating Deep Local Features show that Fisher vectors over frozen deep features beat average and max pooling. The first reports 0.859 vs 0.820 accuracy on TCGA Bladder; the second shows the approach stays robust with little training data. Both use the encoding for retrieval and classification of images, not for auditing a model's own training history.
Measuring data leakage in machine-learning models with Fisher information measures leakage through the Fisher information of a model's parameters. That is practical only for generalized linear models. We move the gradient computation onto a small mixture model over activations, a parameter set of a few thousand rather than billions. This is a different object and does not inherit that paper's privacy guarantees.
One caveat from the feature-extraction literature shapes the design. Frozen embeddings are built to be invariant to surface variation, and this is exactly why mean-pooled representations might wash out a memorization signal. It is also why the second-order statistics, rather than the mean, are the part we test.
Research Program
We build the audit pipeline in four stages, and every component is frozen or fitted without contamination labels.
The first stage builds the reference. We sample roughly ten million tokens of generic web and educational text from The FineWeb Datasets, plus math text from before the base model's training cutoff. We run this text through the base model and record hidden states at about six evenly spaced layers. For each layer we fit PCA to 64 dimensions and a 32-component diagonal Gaussian mixture. A Fisher vector then has 2 × 32 × 64 = 4,096 dimensions per layer, which is small. Because the mixture is fitted only on base-model activations, it is a fixed ruler. Post-trained activations are projected through the same PCA, which is reasonable because post-training from a shared base moves activations comparatively little. We measure that drift explicitly through the fraction of post-trained token states whose posterior mass falls below a threshold. This guards against the degenerate-posterior failure that the Fisher vector literature warns about.
The second stage is encoding. Each benchmark question, prompt only with no generated solution, gets mean-block and variance-block Fisher vectors from both models, and we take their difference $\Delta$. We report results with and without the L2 normalization step, because L2 normalization removes overall magnitude and may erase part of the tightening signal.
The third stage is scoring. In the label-free version, we estimate a mean and a shrinkage covariance of $\Delta$ on a clean null set of questions written after the cutoff. A question's score is its Mahalanobis distance to that null, and we also report a one-sided signed score on the variance block alone, which directly tests the directional prediction. In the oracle version, used only to bound what is achievable, a logistic regression is trained on $\Delta$ with cross-validated known members.
The fourth stage is analysis. We record which layers and which mixture components carry the signal, and whether those components correspond to mathematical notation, problem-statement phrasing, or numbers. If the signal sits in components that respond to specific constants in a problem, that would agree with LaRA's finding that removing key facts disturbs contaminated questions most.
Baselines
The naive reference points are chance (0.5 ROC-AUC) and the output-level loss and Min-K% detectors, run on the same member and non-member questions. The fragility paper reports that after GRPO these lose about 16% AUROC for loss and fall from 89.1% to about 75% for LiRA, and that under chain-of-thought contamination the average across ten detectors is 50–58%. We will reproduce these detectors on our own checkpoints so that the comparison does not depend on numbers transcribed from another setup.
The strongest existing competitor is LaRA, because it is the only cited method designed for exactly this regime and it also uses base versus post-trained internal states. On Eurus-2-7B-PRIME and LIMR, with PRIME-RL and RL-MIA Math as members and AIME 2026 as non-members, LaRA combined with self-critique reports ROC-AUC 0.79 and 31–38% true-positive rate (TPR) at 5% false-positive rate (FPR) (LaRA). We will rerun LaRA on the same models, layers, member and non-member splits, and clean reference set. LaRA uses several extra forward passes per question for its perturbations. We will count its total forward passes and also report our method at a matched budget by adding Fisher vectors of LaRA's paraphrased neighbors, so the gain cannot be attributed to compute. Beating LaRA means higher TPR at 5% FPR with overlapping-free bootstrap intervals over three seeds of our controlled contamination. Matching it with a single pair of forward passes and no perturbation is a weaker but still useful result.
Three ablations isolate the key ingredient. Mean-pooled differencing takes the difference of each question's average hidden state across the two models, scored with Mahalanobis distance against the same null; it isolates whether the mixture matters at all. The mixture log-likelihood difference keeps the mixture but discards the gradient structure, so it tests whether a plain density score suffices. The means-only and variances-only Fisher vector blocks separate first-order from second-order information. A single-model Fisher vector, computed with no base-model differencing, tests whether differencing is what removes the topic and style confound. The proposal's claim is supported only if the variance block beats both mean-pooled differencing and the mixture likelihood difference.
At the benchmark level we also compare against PaCoST. PaCoST asks the model for its confidence that an answer is correct, on original and rephrased questions, and runs a paired significance test. We compare at the benchmark level because PaCoST does not score individual questions. We aggregate our scores per benchmark and ask whether both methods flag the same contaminated benchmarks with p < 0.05.
Why not just…?
Why not just use perplexity or Min-K% on the post-trained model? The fragility paper shows that reinforcement learning shrinks the probability gap between seen and unseen questions, and that chain-of-thought contamination pushes ten such detectors to near chance. These detectors still appear in our tables, because if they recover on our checkpoints the motivation weakens, and we would report that.
Why not just difference the mean-pooled hidden states? This is the most serious threat. It is cheaper, and if contamination simply moves a question's representation, it may work as well. The pathology results (Aggregation Schemes) suggest pooling loses information, but that was a different task. Our ablation settles the question directly. If mean-pooled differencing matches the variance block, we will report that the second-order hypothesis is wrong and recommend the simpler detector.
Why not just run LaRA? LaRA needs a paraphrase and a fact-blanked version of every question. Generating those adds cost and can change a problem's difficulty, the same concern raised about rephrasing in ConStat. LaRA's 31–38% TPR at 5% FPR also leaves most contaminated questions undetected. The compute-matched comparison shows whether our gain comes from a better statistic or from having fewer inputs.
Why not just train a probe on hidden states? A real audit has no labeled contaminated questions. We include the supervised probe only as an upper bound, and the gap between it and our label-free score measures how much the clean-null calibration costs.
Proposed Experiments
Experiment one is the decisive test in the reinforcement-learning regime. We replicate LaRA's setting on Eurus-2-7B-PRIME and LIMR and add our own controlled contamination following the fragility paper. Starting from a public base model, we fine-tune on a randomly chosen half of AIME, AMC, OlympiadBench, Minerva Math and GPQA Diamond questions, then apply GRPO, so membership is known exactly. We report ROC-AUC and TPR at 1% and 5% FPR for every baseline and ablation. The decisive check is the sign test: is the variance block's mean $\Delta$ significantly negative for members and near zero for non-members, at the same layers in both model families?
Experiment two is chain-of-thought contamination, where output detectors fail most completely. We contaminate DeepSeek-R1-Distill-Llama-8B with worked solutions, following the fragility paper's chain-of-thought stage, and ask whether the variance block recovers detection where the output detectors average 50–58% AUROC.
Experiment three is soft contamination. Following Soft Contamination, we fine-tune on semantic duplicates of MuSR questions rather than the questions themselves, then score the originals. Success means the original questions are flagged even though their exact text was never seen, which separates our detector from string matching.
Experiment four tests the confound and the method's limits. We score genuinely clean questions written after the cutoff and drawn from different topics, to see whether topic shift alone produces false positives. We run this with and without base-model differencing. We also report results when the mixture is fitted on a different reference corpus, to measure sensitivity to the reference distribution.
We state one limitation up front. The method needs white-box access to a base model and the model post-trained from it. When no base checkpoint is released, only the single-model variant applies, and experiment four measures how much that degrades the audit.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Soft Contamination Means Benchmarks Test Shallow Generalization
- LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
- XRCE's Participation to ImageCLEF 2008
- Fisher vectors with cascaded normalization for paralinguistic analysis
- Aggregation Schemes for Single-Vector WSI Representation Learning in Digital Pathology
- Species-Agnostic Patterned Animal Re-identification by Aggregating Deep Local Features
- Measuring Data Leakage in Machine-Learning Models with Fisher Information (Extended Abstract)
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models