Paired Hashed Sketches for Contamination That Reinforcement Learning Hides

Reasoning models that have seen a benchmark's test problems during training are hard to catch. Most detectors look for the model being unusually confident on the exact text of a test item. A short round of reinforcement learning after training erases that confidence gap, and the detectors fall to coin-flip accuracy. We propose a different signal, one that needs only sampled text. Ask the model the original question many times. Then ask it a reworded or renumbered version many times. Measure how much the model's solution paths collapse onto one path in each case. A model that memorized a solution should repeat the same intermediate steps and values on the original far more than on the variant. A model that is simply sharp from reinforcement learning should be about equally repetitive on both.

Feature hashing makes this measurement practical. Feature hashing, or "the hashing trick," maps an open-ended set of text features into a fixed-size vector using a hash function instead of a dictionary. The features here are word n-grams, numeric intermediate results, equation fragments, and the order of steps. A second hash supplies a random ±1 sign to each feature, so that colliding features cancel and inner products between hashed vectors remain unbiased. Under that property, one fixed-size vector per question gives an unbiased estimate of the average pairwise similarity among all of that question's samples. This turns "how collapsed are the outputs" into a closed-form statistic with a clean paired test across benchmark items.

Core Claim

Two parts of the claim are distinct. The first is empirical. A paired, text-only measure of how much a model's solution trajectories concentrate, original versus semantically equivalent variant, will separate contaminated from clean benchmarks in the regimes where likelihood-based detectors fail:

  • supervised contamination followed by clean GRPO (a reinforcement-learning method), and
  • contamination with chain-of-thought (written-out reasoning) traces.

Both regimes are documented in On The Fragility of Benchmark Contamination Detection in Reasoning Models. The second part is methodological. Signed feature hashing, as introduced in Feature Hashing for Large Scale Multitask Learning, is the right representation for these trajectory features. The feature vocabulary is open-ended: every new numeric value is a new feature. The estimator also needs unbiased inner products, not just compression.

We do not claim that hashing creates the signal. An exact-dictionary version should perform the same, and we will check this. Hashing is what makes the statistic dictionary-free, fixed-memory, and easy to aggregate over thousands of items and samples.

Relation to Prior Work

Three groups of prior work set up the problem.

Work that motivates it. Generalization or Memorization showed that contamination makes a model's output distribution "peak." Its CDD method measures this through pairwise edit distance among samples. Edit distance is a surface, token-level comparison of single questions. It will miss memorization that survives rewording of the reasoning, and it is not paired against a variant. That pairing matters because reinforcement learning sharpens every output, contaminated or not. On The Fragility of Benchmark Contamination Detection in Reasoning Models is the central negative result. It shows that clipping in GRPO-style training collapses the log-probability gap between seen and unseen items, so likelihood-based detectors fail. A Narrowing Geometry in Contaminated Reasoning shows that contamination leaves an internal low-dimensional signature in hidden activations. This suggests memorization is structural rather than purely lexical, but reading that signature requires access to the model's internals.

Work that supplies the test design. PaCoST pairs original and rephrased questions and runs a paired t-test on the model's confidence, but its confidence measure, P(True), needs log-probabilities. We keep the pairing and replace confidence with a text-only concentration statistic. Proving Test Set Contamination in Black-Box Language Models shows the value of exact, assumption-light statistical tests. We adopt its permutation-test style, though we do not need its ordering assumption.

Work that supplies the representation. Signed hashing comes from Feature Hashing for Large Scale Multitask Learning, including its unbiasedness result and its tail bounds when no single feature dominates. The observation that hashed n-gram spaces tolerate heavy compression comes from Dras et al.. The closest hashing work near contamination is MinHash deduplication in The FineWeb Datasets. That work hashes training documents to remove duplicates. It never audits a trained model, never looks at model outputs, and needs corpus access, which commercial models do not allow.

Research Program

Features. For each sampled completion $y$, we extract a set of features $F(y)$ from three families:

  • word n-grams;
  • normalized numeric literals and equation fragments (for example "17*3=51"); and
  • step-transition pairs linking consecutive intermediate values.

The third family is the one we expect to carry memorized trajectories through paraphrase. Each feature $f$ is mapped to bucket $h(f)\in\{1,\dots,B\}$ with sign $\xi(f)\in\{-1,+1\}$. The completion's sketch $\phi(y)$ is the sum of $\xi(f)e_{h(f)}$ over its features, where $e_k$ is the $k$-th unit vector. We normalize $\phi(y)$ to unit length.

Statistic. For a question with $n$ sampled completions $y_1,\dots,y_n$, the average pairwise similarity is

$$S = \frac{\lVert \sum_{j} \phi(y_j)\rVert^2 - \sum_j \lVert\phi(y_j)\rVert^2}{n(n-1)}.$$

This needs only the running sum of sketches. Because signed hashing preserves inner products in expectation, it estimates the same quantity an exact-dictionary representation would. Its distortion is bounded as in Feature Hashing for Large Scale Multitask Learning.

For benchmark item $i$, we compute $S$ on samples from the original question ($S^{\text{orig}}_i$) and from a variant ($S^{\text{var}}_i$). The per-item difference is

$$d_i = S^{\text{orig}}_i - S^{\text{var}}_i.$$

The benchmark-level test is a one-sided sign-flip permutation test on the mean of $d_i$. The item-level detector score is $d_i$ itself, which we evaluate by AUROC (area under the ROC curve).

Variants. We use two kinds:

  • Rephrasings, produced by a helper model as in PaCoST, which keep the answer the same.
  • For math, numeric perturbations: the same problem template with changed numbers.

Numeric perturbations also allow a carryover probe. We build a sketch of the original item's reference solution, then measure its inner product with the model's outputs on the perturbed item. A contaminated model should leak the original problem's intermediate values into solutions where those values are wrong.

Analysis. We will measure:

  • how power scales with the number of samples $n$ and the number of buckets $B$;
  • which feature family carries the signal; and
  • whether the signal persists over successive epochs of GRPO, in the same staged protocol as the fragility paper.

Baselines

Naive reference points. The do-nothing reference is chance, 50% AUROC. A slightly stronger naive reference is the raw accuracy gap between original and variant questions. This is the logic behind the HumanEval versus HumanEval_R comparison in Generalization or Memorization, which reported a Leak Ratio of about 41.5% for ChatGPT on HumanEval. The accuracy gap is text-only and costs nothing extra, so our statistic must beat it at matched sample budgets. If it does not, the trajectory features add nothing.

Strongest likelihood-based competitor. This is LiRA, with loss thresholding and Min-K% Prob alongside it, run with full log-probability access. On The Fragility of Benchmark Contamination Detection in Reasoning Models reports LiRA at 89.1% AUROC on SFT contamination before reinforcement learning. After one GRPO epoch it drops by about 14%, and reference-free methods approach about 50% AUROC. Under chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B, the ten tested detectors averaged 50–58% AUROC. We reproduce these on the same contaminated checkpoints. Beating them means our text-only statistic exceeds 70% AUROC after GRPO, or in Stage II, while staying within a few points of LiRA before reinforcement learning. This is a fair comparison that favors the baselines, since they get strictly more access.

Strongest output-distribution competitor. This is CDD from Generalization or Memorization. It is the closest black-box method and is reported to improve on n-gram baselines by 21.8% to 30.2% in Accuracy and AUC. We run it on the same samples, with the same $n$ and the same temperature, so any gain comes from features and pairing rather than sampling budget. We also run a paired version of CDD, which applies its edit-distance score to both the original and the variant and takes the difference. This separates the contribution of pairing from that of trajectory features.

PaCoST. We run PaCoST on the same rephrasings, as the paired confidence-based alternative.

White-box upper reference. On GSM8K and MSCalc, we use the eigenspectrum concentration measure from A Narrowing Geometry in Contaminated Reasoning. That paper reports detection only indirectly, through an 11.03% consistency improvement for its suppression method. We found no reported Lacuna result for its detection AUROC under reinforcement learning, so we compute it ourselves.

Ablations. These isolate the key ingredients:

  • an exact-dictionary version of our statistic, which should match the hashed one, confirming hashing costs nothing;
  • surface n-grams only, without numeric or step-transition features;
  • the unpaired statistic $S^{\text{orig}}$ alone; and
  • a sweep over $B$ from $2^{12}$ to $2^{22}$.

All methods share the same models, items, variants, and generation budget. Any hyperparameter tuning, such as $k$ in Min-K% or the threshold in CDD, uses a held-out contaminated/clean split.

Why not just…?

Why not just use CDD's edit distance? CDD measures how peaked outputs are on the original question alone. Reinforcement learning makes every output more peaked, so an unpaired peakedness score should rise on clean items too. The file does not list which ten detectors the fragility paper tested, so we cannot cite a CDD failure under reinforcement learning. The paired-CDD baseline settles this directly. If paired edit distance matches our statistic, then pairing was the whole contribution, and the trajectory features and hashing are unnecessary. We would report that outcome as the finding.

Why not just embed outputs with a dense sentence encoder and measure cosine concentration? This is the most serious threat. A dense encoder also yields a fixed-size vector, and its mean-vector trick works the same way. Our bet is that dense encoders capture topic and wording, and so assign high similarity to any two correct solutions of the same problem. Memorization, by contrast, shows up in specific intermediate values and step order, which hashed literal features capture and dense embeddings blur. We include a dense-embedding version, paired in the same way, as a baseline. The numeric-perturbation carryover probe is where the two should differ most. If dense embeddings match us, we report the hashed sketch only as the cheaper, dictionary-free, model-free option.

Why not just compare accuracy on originals versus variants? Accuracy is one bit per sample. A model contaminated with reasoning traces may solve both original and variant correctly while still reproducing the memorized path, and the fragility paper's Stage II contamination raised performance by 11.7%. Trajectory concentration should therefore detect contamination with fewer items. We test this by plotting detection power against the number of items for both statistics.

Why not just read the internal geometry? The narrowing signature requires model weights and activations. It is a white-box upper bound, not a substitute for auditing models behind an API. If our text-only statistic tracks that signature across GRPO epochs, it is the black-box counterpart that commercial models need.

Proposed Experiments

Main experiment. We replicate both stages of the fragility protocol on AIME, AMC, GPQA Diamond, OlympiadBench, and Minerva Math:

  • Stage I: contamination through supervised fine-tuning, then 0, 1, and 2 epochs of clean GRPO.
  • Stage II: chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B.

At each checkpoint we report item-level AUROC and benchmark-level p-values for every baseline.

Clean controls. We calibrate false positives on WMDP, which postdates the models PaCoST evaluated, and on CodeForces2305, which postdates ChatGPT's cutoff. On these clean sets, the benchmark-level test must not reject at p<0.05 more than 5% of the time.

Exact versus implicit leakage. On DETCON we test whether the statistic handles both exact and rephrased leakage. We also compare it against the HumanEval/HumanEval_R gap.

Decisive checks. The proposal fails in any of the following cases:

  • The post-GRPO AUROC of our paired statistic does not exceed the reported 50–58% range by a clear margin.
  • Paired CDD or paired dense embeddings perform as well.
  • The exact-dictionary and hashed versions diverge at moderate $B$.

The proposal is supported in two cases:

  • The step-transition and numeric features carry most of the signal while surface n-grams do not.
  • The carryover probe shows original-solution values appearing in perturbed-item outputs, only for contaminated models.

Together, these would show that trajectory memorization survives reinforcement learning even after the likelihood gap is gone.