A language model that has memorized a benchmark question answers it because it recognizes the exact wording. A model that has learned the skill answers it because it understands the problem. Behaviorally the two can look identical, and recent work shows that standard reinforcement learning after pretraining makes the usual statistical evidence of memorization vanish. This proposal measures the difference inside the model. For each benchmark item we build a family of rewordings that preserve its logic. We then train a small probe on the model's hidden states that splits each item's representation into two parts: a "logic code" shared by the whole family, and a "surface code" specific to each wording. The surface code is placed under an adjustable information budget, the knob that governs the familiar trade-off in variational autoencoders between clean, factorized latents and faithful reconstruction. The audit statistic is the surface rate. It asks how many bits of wording-specific information the probe must spend to predict the model's confidence in its answer on the original phrasing, compared with the rewordings. If the model has memorized an item, its original wording should be atypically expensive. If the model has learned the skill, the original wording should be exchangeable with its siblings.
This matters because the most worrying contamination regime now has no working detector. On The Fragility of Benchmark Contamination Detection in Reasoning Models finds that one epoch of clean GRPO, a policy-gradient reinforcement learning method with clipped updates, cuts the AUROC of the LiRA membership attack by about 14% and of loss-based detection by about 16%. When contamination arrives as chain-of-thought training data, ten detectors fall to 50–58% AUROC even though pass@1 is inflated by 11.7% on DeepSeek-R1-Distill-Llama-8B. Those detectors all read output probabilities, and clipping equalizes exactly those. Our bet is that clipping compresses the gap in output log-probability without erasing the model's internal dependence on the canonical wording. We also bet that a bottlenecked, factorized probe can measure that dependence more reliably than raw activations or behavior alone.
Core Claim
Contamination is a property of the dependence between a model's answer and the surface form of the question. It is not a property of how likely the question looks. That dependence can be measured as a rate, meaning the bits of wording-specific information needed to explain the model's answer confidence. The measurement should keep working when likelihood-based signals have been cloaked. The new element is not "disentangle logic from surface", which A²RBench already probes behaviorally through symbolic remapping (a 17.7% accuracy drop for top models). The new element is to turn the disentanglement-reconstruction trade-off into a measuring instrument. We sweep the surface-code budget and test whether the canonical phrasing sits off the curve traced by its own logic-preserving variants. This is a within-item exchangeability test carried out in representation space.
How the Two Literatures Fit Together
The contamination literature supplies the problem and the statistical template. Proving Test Set Contamination in Black-Box Language Models tests whether a dataset's canonical ordering is more likely than random permutations. Under no contamination the orderings are exchangeable, and the authors detect data seen only 2–4 times. We keep that exchangeability logic but exchange wordings of one item instead of orderings of a dataset, and we read hidden states instead of log-probabilities. ConStat defines contamination as non-generalizing performance and calibrates against rephrased reference benchmarks and a fleet of reference models. It is our closest conceptual relative. However, it works at the benchmark level and needs a reference fleet. We work at the item level and need only the subject model, a variant generator, and clean calibration items. Data Contamination Calibration for Black-box LLMs perturbs word order and reads the spread of token confidences. Like LiRA and Min-K%, it is an output-probability signal of the kind the Fragility paper shows clipping suppresses. Dynamic generators such as DyVal and A²RBench enter here as tools rather than competitors. Their programmatic structure gives us families of variants whose logic is provably shared.
The disentanglement literature supplies the method and one key constraint. Group-level factorization, in which a content code is shared across examples known to share a factor and a style code is free per example, follows Disentangling Factors of Variation with Cycle-Consistent Variational Auto-encoders. We reuse its idea of a group-pooled content code, with an equivalence class of rewordings as the group. Rate-Regularization and Generalization in Variational Autoencoders motivates reading memorization off the rate axis. It shows that high-capacity autoencoders memorize specific factor combinations and that the regularization strength at which test reconstruction is best behaves differently from low-capacity models. This tells us a single fixed budget is the wrong measurement and the whole curve is informative.
The same literature also warns of a confound. A heavily regularized probe can fail to predict well for its own reasons, which would show up as spurious distortion. Two papers supply the fixes. DynamicVAE uses a proportional-integral controller to hold the KL term at a chosen target, which we use to pin the surface code to exact rates so that canonical and variant are compared at matched budgets. Training β-VAE by Aggregating a Learned Gaussian Posterior with a Decoupled Decoder freezes a heavily regularized encoder and retrains an unconstrained decoder, recovering full fidelity. We use this so that measured distortion reflects the subject model and not probe underfitting.
Two theory papers motivate but do not supply the method. Hallucination is a Consequence of Space-Optimality gives a rate-distortion account of memorizing unstructured facts and explicitly leaves structured content open. Our measurements on structured reasoning items are a first empirical look at that regime. Bayesian Perspective on Memorization and Reconstruction warns that membership attacks conflate domain-level prior knowledge with memorization of a specific sample. Our within-item comparison is designed to cancel that confound, because canonical wording and variants carry the same domain knowledge.
Research Program
Let $h_{i,k}$ be the subject model's hidden state at the final prompt token for variant $k$ of item $i$, with $k=0$ the canonical benchmark wording. Let $y_{i,k}$ be the model's log-probability of its final answer on that variant. The probe encodes a logic code $z_c$ by pooling over all variants of item $i$ and a surface code $z_s$ from $h_{i,k}$ alone. It is trained on many items to minimize
$$ \mathcal{L} = \mathbb{E}\left[-\log p(y_{i,k} \mid z_c, z_s)\right] + \beta \, D_{\mathrm{KL}}\left(q(z_s \mid h_{i,k}) \,\|\, \mathcal{N}(0,I)\right), $$
where $\beta$ sets the surface budget and the KL term is the surface rate. We train the encoder at a ladder of controlled target rates using the DynamicVAE controller. At each rate we refit an unconstrained decoder on the frozen encoder. For each item and rate $R$, the statistic $S_i(R)$ is the canonical wording's prediction loss minus the median loss over its variants. Aggregated across items and across the rate ladder, this yields a per-item score and a benchmark-level one-sided test.
The test is calibrated on items known to be clean, using the same variant generator. This calibration is the crux. LLM-written paraphrases differ stylistically from human originals, so canonical wordings can look atypical for reasons unrelated to memorization. We handle this in two ways. On DyVal and A²RBench the canonical item is itself drawn from the generator, so it is exactly exchangeable with its variants under the null hypothesis. On GSM8K we obtain the null from the half of test items that we deliberately do not inject, rewritten with the same entity-and-phrasing remapper that keeps the equation graph fixed.
The analytic program asks three questions. Where along the rate ladder does the canonical–variant gap open? Does that location shift with duplication count? Does reinforcement learning move the gap or merely shrink output log-probability spread? The last question is the direct test of the cloaking hypothesis.
Baselines
The naive reference point is loss-based detection, which flags items with unusually low loss. The strongest output-probability competitor is LiRA. On the Fragility suite (AIME, AMC, GPQA Diamond, OlympiadBench, Minerva Math), LiRA reaches 89.1% AUROC before reinforcement learning, drops about 14% after one GRPO epoch, and the ten-detector family averages 50–58% AUROC under chain-of-thought contamination. We rerun both, together with Min-K% and PAC, on our own contaminated checkpoints so that data, model, and item splits are identical. All detectors receive the same tuning budget, measured in hyperparameter trials on a held-out calibration model.
We also run the Sharded Rank Comparison Test, the strongest exchangeability-based method, at matched duplication counts, since its reported sensitivity is detection at 2–4 duplications. ConStat is the strongest performance-based method; it reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models. We run it with rephrased reference sets built by our own variant generator and a reference fleet of uncontaminated checkpoints trained with the same compute. We found no reported result on Lacuna for ConStat on GRPO-cloaked or chain-of-thought-contaminated models, so that comparison is new.
The baseline we consider most dangerous is the behavioral variant gap: the item-level difference between the model's answer confidence on the canonical wording and its mean confidence on the variants, with the same variants and no probe. Two further baselines isolate the key ingredients. The first is raw-activation atypicality, the Mahalanobis distance of $h_{i,0}$ among $\{h_{i,k}\}$, which tests whether the bottleneck matters at all. The second is a supervised activation membership probe trained on a shadow model with known members and transferred to the target.
Our ablations remove one ingredient at a time. We drop the shared logic code, which leaves a plain bottleneck. We use a single fixed $\beta$ instead of the rate ladder. We skip the decoupled decoder refit.
Beating the baselines means two things. In the post-GRPO and chain-of-thought regimes, item-level AUROC must clearly exceed the 50–58% band. At a matched false-positive rate on clean models, the method must also beat the behavioral variant gap and raw-activation atypicality given the same number of variants per item.
Why not just…?
Why not just measure the accuracy or confidence drop on reworded versions? This is the real threat, and it is the baseline we most expect to be competitive. In the pure supervised fine-tuning regime it may match us, and we will report that if so. Our reason to expect a gap under cloaking is that the behavioral signal reads the very output confidences that clipping compresses. In addition, chain-of-thought contamination teaches reasoning steps that transfer to nearby variants, so the canonical–variant behavioral gap narrows. The surface-rate test asks a different question: how many bits of wording-specific internal information explain the model's confidence, even when that confidence is similar across wordings. If the behavioral gap equals our method under GRPO and chain-of-thought contamination at matched variant counts, the representation machinery is not earning its cost and the project's central claim fails.
Why not just use ConStat? ConStat is robust to rephrasing by design, but it scores whole benchmarks and needs a fleet of reference models. Its definition of contamination as inflated, non-generalizing performance also means that inflation which transfers to the rephrased reference set can go unflagged, and the Fragility paper suggests chain-of-thought contamination produces exactly this kind of transfer. We run ConStat head-to-head on identical checkpoints. If it detects the chain-of-thought-contaminated models at the benchmark level as well as we do, our remaining contribution is item-level attribution, and we would say so.
Why not just train a membership classifier on hidden states? Auditors never have labeled members for the target model, so such a classifier must be trained on a shadow model and transferred. That classifier also inherits the confound flagged by Bayesian Perspective on Memorization and Reconstruction, where familiar domain content is mistaken for a memorized sample. Our statistic compares each item with its own logic-equivalent siblings, so domain familiarity cancels. The shadow-probe baseline measures how much this matters in practice.
Why not just switch to dynamic benchmarks? DyVal is confined to formal domains. A²RBench reports that its authoring model simplifies 3D rules, which narrows what can be generated. Neither approach helps audit the static benchmarks such as ARC, GSM8K, and MMLU that are still used to rank models. We use the generators as sources of variants rather than replacements for those benchmarks.
Proposed Experiments and Decisive Checks
The main experiment uses controlled contamination following the Fragility protocol. Starting from an open base model, we inject half the items of GSM8K, a DyVal-generated arithmetic and reachability set, A²RBench items, and the Fragility math and science suite. We inject at 1, 2, 4, and 10 duplications to mirror the canary study of Proving Test Set Contamination in Black-Box Language Models. We then produce three checkpoints per setting: fine-tuning only, fine-tuning followed by one epoch of clean GRPO, and chain-of-thought contamination. We report item-level AUROC and benchmark-level detection power at a fixed false-positive rate measured on matched clean checkpoints, for every baseline and ablation above.
The first decisive check is the cloaking test. Moving from the fine-tuning checkpoint to the GRPO checkpoint should leave the surface-rate AUROC roughly intact while loss and LiRA degrade as the Fragility paper reports. If our score drops just as much, the hypothesis that clipping leaves internal dependence on wording intact is false.
The second decisive check is dose-response. The rate at which the canonical–variant gap opens should increase with duplication count, and it should be absent on the exchangeable DyVal null. A gap on clean generator-drawn items would indicate miscalibration.
The third check attributes credit through the ablations. The shared logic code and the rate ladder should each contribute beyond raw-activation atypicality. If they do not, the disentanglement framing is decorative and we would report the simpler statistic instead.
Finally, a field audit on open-weight models tests agreement with prior findings. We will check whether the method agrees in direction with the reported contamination of Mistral-7B on AI2-ARC ($p = 0.001$ from the rank test) and with ConStat's estimate of an approximately 8.2% performance inflation for Mistral-7B-v0.1 on GSM8K. For ARC we build variants by paraphrasing and shuffling answer options, and we flag the weaker null calibration this entails. Agreement would be supporting evidence, not proof, because we have no ground truth for these models.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- Proving Test Set Contamination in Black-Box Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Data Contamination Calibration for Black-box LLMs
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Disentangling Factors of Variation with Cycle-Consistent Variational Auto-encoders
- Rate-Regularization and Generalization in Variational Autoencoders
- DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
- Training β-VAE by Aggregating a Learned Gaussian Posterior with a Decoupled Decoder
- Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing
- Bayesian Perspective on Memorization and Reconstruction