Choosing Codebook Size for Contamination Audits of Reasoning Models

When a language model has seen a benchmark's test questions during training, its high score may come from memory rather than reasoning. Today's tools for catching this mostly read the model's output probabilities, and they break in a specific way. After even one round of clean reinforcement learning, the probability gap between seen and unseen questions disappears, and detectors fall to near chance. Yet the model's internal computation on memorized questions still appears to behave differently. It settles into the same rigid internal path every time it is asked. We propose to measure that rigidity by snapping the model's hidden states onto a small dictionary of discrete codes, called a codebook, and then checking how similar the resulting code sequences are across repeated samples of the same question. The design question that decides whether this works is how large the codebook should be. That question is the same capacity tradeoff studied in discrete representation learning, now applied to measurement rather than to modeling.

The core prediction is a U-shape. If the codebook is too small, every reasoning trace maps to nearly the same codes, so seen and unseen questions look equally "converged." If it is too large, every trace is unique, and convergence on memorized items is lost in noise. The best codebook size should grow with the audit's sample budget, both the number of sampled traces per question and the number of questions, in a way we can predict and calibrate. The deliverable is a gray-box audit for open-weight models that comes with a rule for choosing the codebook size and a calibrated estimate of its statistical power, tested specifically in the regime where reinforcement learning hides contamination from likelihood-based detectors.

Core Claim

Three findings motivate the claim. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates models through supervised fine-tuning and then applies clean GRPO, a common reinforcement-learning recipe for reasoning. It finds that the PPO-style clipping step shrinks the log-probability gap, so detectors that work after fine-tuning drop to roughly 50–58% AUROC. Detecting RLVR Training Data via Structural Convergence of Reasoning shows that reinforcement learning with verifiable rewards collapses reasoning onto rigid, low-diversity paths for seen prompts, and it detects this by comparing sampled output texts with edit distance (Min-kNN Distance). LaRA finds a related signature inside the network: hidden states of contaminated prompts are stiff under paraphrase and compress into few directions. Together these suggest that the signal survives reinforcement learning internally even when it vanishes from output likelihoods, and that the text surface only partly reveals it.

Our claim is that discretizing hidden-state trajectories turns this internal rigidity into a count statistic. Unlike a raw continuous distance, that statistic has an interpretable bias–variance tradeoff governed by codebook size $K$. The tradeoff shapes how we interpret detection results. When Is Benchmark Contamination Detectable? shows that detection power is governed by $\alpha\rho\sqrt{m}$. Here $\alpha$ is the fraction of contaminated items, $\rho$ is the behavioral separability between seen and unseen items, and $m$ is the number of audited items. With a codebook, separability becomes a function of $K$. We write

$$\text{signal}(K) \approx \alpha\,\rho_K\sqrt{m}, \qquad \hat\rho_K = \rho_K + O\!\left(\tfrac{K}{N}\right),$$

where $\rho_K$ is the true separability visible at resolution $K$, which grows with $K$ and then saturates. $\hat\rho_K$ is its estimate from $N$ sampled traces per item. The $K/N$ term reflects the usual upward bias of plug-in divergence estimates, which scales with the number of free cells relative to the number of samples. The result is an interior optimum $K^\*$ that should rise as $N$ grows.

This is the measurement-side analog of the law in Task-Driven Discrete Representation Learning. That paper proves that downstream sample complexity grows quadratically with codebook size and shows that discrete bottlenecks win in low-data regimes. We use the same logic for choosing an auditor's resolution, not for training a model.

The proposal deliberately does not insert a codebook into the model to block memorization. That route is the obvious reading of this intersection, and it has two problems. It requires retraining a model with a new bottleneck, and its effect is confounded with simple capacity loss. Direct circuit ablation, as in Spurious Rewards Paradox, already shows how to suppress the memorization pathway in Qwen models without any discretization. Our auditor leaves the model untouched.

Research Program

The audit has four parts.

First, we pick an audit layer. Default candidates are the mid-to-late layers that Spurious Rewards Paradox identifies as the point where computation branches toward retrieval in Qwen2.5 (layers 18–20). We also run a sweep across layers for other model families.

Second, we fit a codebook on clean data only. Hidden states come from the same model run on questions it cannot have seen: freshly generated items from RandomCalculation (Reasoning or Memorization?), post-cutoff items such as LiveMathBench, or a held-out half of the benchmark in controlled experiments. We project these states to a few dimensions and quantize them with Finite Scalar Quantization, introduced in Finite Scalar Quantization: VQ-VAE Made Simple. That method replaces a learned dictionary with fixed rounding per dimension and uses every code by design up to $2^{16}$ codes. We use it because a learned codebook's dead codes would silently change the effective $K$ and confound the sweep. Its low-dimensional design, normally a limitation, suits a measurement probe.

Third, we score each question. For each audited question we sample $N$ reasoning traces, map each token's hidden state to a code, and compute the average of the $k$ smallest pairwise edit distances between code sequences. This is Min-kNN Distance computed on internal codes instead of output text.

Fourth, we test at the benchmark level. We compare the distribution of per-item scores on the audited benchmark against clean twin items using a permutation test on counts. We then predict the test's power from $\hat\rho_K$ using the planning approach of When Is Benchmark Contamination Detectable?, so that a "no contamination found" verdict comes with a stated power instead of silence.

The analysis component is the $K$ sweep. We vary $K$ from $2^4$ to $2^{16}$, crossed with $N \in \{4, 8, 16, 32\}$ and several audit sizes $m$. We locate $K^\*$ in each cell and test whether it shifts with $N$ as the bias term predicts.

Baselines

The naive reference is chance (0.5 AUROC). A second naive reference is the partial-prompt completion test of Reasoning or Memorization?: truncate the question and check whether the model still answers correctly. That paper reports Qwen2.5-Math-7B reaching nearly 40% accuracy on MATH-500 from only the first 40% of each prompt, against near zero for Llama-3.1-8B. This test catches verbatim memory but has no stated power and does not address reinforcement-learning cloaking.

Likelihood detectors are represented by LiRA and Min-K%++. Min-K%++ normalizes token log-probabilities and averages the lowest outliers, without a reference model. LiRA is the strongest detector in the fragility study, which reports 89.1% AUROC after SFT contamination and a drop of about 14% after one clean GRPO epoch. We found no reported result on Lacuna for Min-K%++ on RL-contaminated math benchmarks. We will run both on identical contaminated checkpoints. Beating them means maintaining clearly higher AUROC after the GRPO step. We do not expect to beat LiRA before it.

The strongest black-box competitor is text-level Min-kNN Distance, which reports an average AUC of 0.70 across 1.5B–32B models (Detecting RLVR Training Data via Structural Convergence of Reasoning). It is the closest method to ours. We will compute it from exactly the same $N$ sampled traces our auditor uses, so the sampling cost is identical and the only difference is whether distances are measured on text or on codes.

The strongest gray-box competitor is LaRA, which reports ROC-AUC 0.79 and 31–38% true positive rate at 5% false positive rate when combined with Self-Critique on PRIME-RL, AIME 2026, and RL-MIA Math (LaRA). We will run it on the same models, layers, and splits. Our method claims success only if it matches LaRA's true positive rate at 5% false positive rate without a clean reference model, or exceeds it with one.

The key ablation is continuous Min-kNN. It uses the same projected hidden states, the same layer, and the same $N$, but replaces code edit distance with a cosine-based trajectory distance (dynamic time warping). Every method gets the same clean data for calibration and the same tuning budget, which is a grid over layer and $k$, with $K$ selected only on a validation split.

Why not just…?

Why not just use procedurally generated benchmarks and stop auditing static ones? Generators such as BeyondBench, DyVal, and RandomCalculation keep future evaluations clean. They cannot say whether a reported MATH-500 or AIME number was inflated, and they cover mainly formal, rule-based domains. We use them as the clean reference our auditor needs, so the two approaches are complementary rather than competing.

Why not just read output likelihoods with Min-K%++ or LiRA? The fragility study shows these fall to roughly 50–58% AUROC after clean reinforcement learning or reasoning distillation. That is exactly the regime this proposal targets. Our experiments rerun them on the same checkpoints, so if they recover, the result will show it.

Why not just use text-level Min-kNN, which needs no weights? This is a real threat. If reinforcement learning collapses the text as strongly as it collapses internal states, codes add access requirements without adding signal. The matched-sampling comparison settles it. We expect a gain only where traces differ in wording but follow the same internal path. If the gain is under a few AUROC points, we will report that the black-box method suffices.

Why not just use continuous hidden-state distances and skip the codebook? This is the most serious alternative, and it is the main ablation. If continuous distances match code distances at their best $K$, the remaining contribution is narrower. It reduces to count-based power calibration and a $K$-selection rule, which continuous distances do not offer as directly. We will state that outcome plainly rather than claim discretization helps detection.

Proposed Experiments

The controlled setting follows the fragility study's protocol. We contaminate Qwen2.5 and Llama-3.1 models at 7–8B scale by fine-tuning on half of AIME, AMC, OlympiadBench, and MATH-500, then apply one and three epochs of clean GRPO. The held-out halves serve as non-members. Item-level AUROC and true positive rate at 5% false positive rate are measured at every stage. A second setting reuses LaRA's RL-contamination splits to compare directly with its published numbers. A natural-contamination check compares Qwen2.5-Math-7B against Llama-3.1-8B on MATH-500, with LiveMathBench as the clean twin. Here the partial-prompt evidence gives a known positive (Qwen) and a known negative (Llama).

Four checks decide the outcome. The first is whether AUROC against $K$ shows an interior peak at fixed $N$. A monotone curve would falsify the capacity-tradeoff framing. The second is whether the peak $K^\*$ rises with $N$, as the $K/N$ bias term predicts. The third is whether the code auditor beats text Min-kNN and LiRA after the GRPO step under matched sampling. The fourth is whether power predicted from $\hat\rho_K$ matches empirical power across contamination fractions $\alpha \in \{0.05, 0.1, 0.25\}$, with a target of agreement within the 0.785–0.92 band that When Is Benchmark Contamination Detectable? reports when its planner aims for 0.80. A null result on the second and third checks would be informative in itself: it would show that the internal rigidity LaRA reports carries no information beyond what sampled text already reveals.