A reward model is a network that reads a prompt and a response and outputs a single score. It is trained on pairs in which one response was preferred over another, and it is judged on held-out pairs by how often it scores the preferred response higher. Benchmarks such as RewardBench are public, static collections of (prompt, chosen, rejected) triples, and their authors warn that future reward models may be trained directly on them. Yet no one can currently check whether a released reward model has seen a given benchmark. Existing contamination detectors read token probabilities, or look for rigid reasoning traces in generated text. A reward model produces neither. It returns one number per response.
We propose a black-box audit built on one observation. A model that was trained on a specific triple has learned about those exact strings, not only about their quality. So its score for the memorized chosen response should sit above its scores for meaning-preserving rewrites of that response, and its score for the memorized rejected response should sit below its scores for rewrites of that one. A clean model has no reason to treat the original and its rewrites differently beyond the stylistic change the rewriting itself introduces. We measure that stylistic change on freshly built control pairs and subtract it out. Scoring the chosen and rejected sides separately also lets the audit say how an item leaked, not only whether it did. If both sides are displaced, the full triple was in the preference-training data. If only the chosen side is displaced, the prompt and winning answer were probably seen during pretraining, with the rejected distractor never seen.
Core Claim
Contamination of a pairwise preference benchmark leaves a signature in a reward model's scores that is specific to each side of the pair and survives the model's known biases toward length and style. That signature can be isolated with a difference-in-differences test that needs only query access to the reward model. The test borrows an idea from pairwise learning. Comparisons made within a batch cancel shared offsets, as in Multi-task Bioassay Pre-training for Protein-ligand Binding Affinity Prediction, where within-assay comparisons remove laboratory-specific baselines. Here the "batch effect" is the reward shift that rewriting alone causes, and it is estimated on control items that cannot have been seen.
Concretely, for benchmark item $i$ with prompt $x_i$ and side $s \in \{+,-\}$ (chosen or rejected), let $\bar r(x_i, \tilde y_i^{s})$ be the reward model's mean score over $K$ meaning-preserving rewrites of $y_i^s$. The side residual is
$$e_i^{s} = r(x_i, y_i^{s}) - \bar r(x_i, \tilde y_i^{s}), \qquad D_i = e_i^{+} - e_i^{-},$$
where $r$ is the audited model's score and $D_i$ measures how much more the model separates the original pair than it separates their rewrites.
The audit compares the distribution of $D_i$, and of $e^+$ and $e^-$ separately, on benchmark items against the same quantities on control items. Control items are new triples built from post-cutoff prompts with the same generation and labeling recipe, and they are rewritten by the same paraphraser. Rewriting may shift scores for length or style reasons. That shift is the same for benchmark and control items and cancels in the comparison. Only memorization of the exact strings survives.
Relation to Prior Work
The closest method is PaCoST, which runs a paired t-test on a language model's confidence for original versus paraphrased benchmark questions. We reuse its paired logic, with three changes. We apply it to a scalar reward rather than a language model's confidence. We split it by side of the pair. And we calibrate against rewrite-matched controls rather than assuming paraphrasing is neutral. That last change matters because reward models are known to reward length and formatting (RewardBench), so a paraphraser that shortens text will move scores on its own.
Two recent detectors motivate the proposal by showing that likelihood-based methods fail after training on rewards rather than on text. Self-Critique reports perplexity and Min-K% at 0.42–0.53 AUC on RL-trained models. Min-kNN Distance detects rigid, repeated reasoning structure in sampled outputs. Both audit a model that generates text on verifiable math and logic tasks. Both leave preference-trained scorers untouched, and Self-Critique explicitly notes that open-ended dialogue is out of scope. LaRA reads hidden-layer geometry, which requires weights. Our audit needs only scores.
On the preference side, Preference Leakage shows that LLM judges inflate win rates for students of related models, driven by style rather than substance. It does not study scalar reward models trained with the Bradley-Terry model, the standard model that turns a difference in two scores into a preference probability. It also does not audit whether benchmark triples were trained on. Its finding is a constraint on our design. Because related-lineage style preference exists, a style-only score shift could masquerade as contamination. The rewrite-matched controls are there to absorb exactly that.
RewardBench 2 defends against leakage by sourcing about 70% of prompts from unreleased WildChat interactions. It finds scores about 20 points lower than on the original RewardBench. It offers no audit for when those prompts eventually leak.
Theory constrains and sharpens the design in two places. What Does Preference Learning Recover from Pairwise Comparison Data? shows that Bradley-Terry fitting is exact only when chosen and rejected responses are conditionally independent. Otherwise it fits the closest Bradley-Terry model in the KL-divergence sense. Asymmetric exposure, where the chosen response is seen and the rejected one is not, is a concrete case of coupled pairs. It predicts displacement on one side only, which the side residuals test directly. When Is Benchmark Contamination Detectable? shows that the power of any audit scales with $\alpha\rho\sqrt{m}$. Here $\alpha$ is the fraction of items contaminated, $\rho$ is how separable contaminated behavior is from clean behavior, and $m$ is the number of audited items. We will estimate $\rho$ for reward models empirically, which turns that bound into a minimum detectable contamination fraction for RewardBench-sized audits.
Research Program
The work has three parts.
The first is a controlled contamination testbed. We train Bradley-Terry reward models on clean base mixtures drawn from UltraFeedback, HH-RLHF and SHP. We use two base-model sizes, because the Preference Leakage and DCR papers disagree on whether small or large models absorb leakage more. Into these mixtures we inject RewardBench triples at fractions $\alpha \in \{0, 1, 5, 20, 50\}\%$, under four exposure modes:
- Full triple: the complete triple is added to reward-model training.
- Rephrased triple: a rewritten version of the triple is added instead, following the recipe in Rethinking Benchmark and Contamination with Rephrased Samples.
- Chosen-only: the prompt and chosen response are added only to continued pretraining of the base model before reward training.
- Clean: no injection.
Items are split at random into injected and held-out halves, which gives ground truth for each item.
The second part is the audit itself. The components are:
- Rewrites: $K$ meaning-preserving rewrites per response from two different paraphrasers. For math and code responses, the rewrite is constrained to preserve final answers and code behavior, and is filtered by execution or answer matching.
- Controls: triples built from post-cutoff WildChat-style prompts with the original benchmark's construction recipe.
- Scores: item-level scores $D_i$, $e^+_i$ and $e^-_i$.
- Dataset-level test: a rank test of benchmark items against control items.
- Exposure classifier: assigns each flagged item to full-triple, chosen-only, or no exposure from the pair $(e^+, e^-)$.
We also extend the audit to RewardBench 2's Best-of-4 format, comparing the chosen response's residual with the residuals of its three distractors.
The third part is calibration and application. From the testbed we fit $\rho$ as a function of exposure mode and model size. We then report the smallest $\alpha$ each detector can flag at 5% false positives on RewardBench-sized and RB2-sized audits. Finally, we run the audit on public reward models from the RewardBench leaderboard.
Baselines
The naive reference is raw margin or accuracy. It flags an item as contaminated when the model's margin $r(x,y^+)-r(x,y^-)$ is large, and flags a dataset when benchmark accuracy exceeds accuracy on the control items. It is what an auditor would do with no method at all. It confounds difficulty with exposure, so beating it on item-level AUC shows that the rewrites carry information. It runs on the same models and items with no tuning.
The strongest existing competitor is PaCoST applied to reward margins (PaCoST). It runs a paired test of original-pair margin against rewritten-pair margin, with no side split and no control calibration. It is the closest published idea and needs the same black-box access, which makes it the fairest comparison. It will use the identical paraphrasers, the same $K$, and the same items, so the only differences are our side decomposition and control subtraction.
We also include Min-K%++ on the reward model's underlying language model (Min-K%++), scoring the prompt concatenated with the chosen response. It applies only when base weights are available. It is the strongest likelihood-based detector in the file and is the natural choice for the chosen-only exposure mode, where the text really was seen during pretraining.
No reported result was found on Lacuna for any detector auditing reward models or pairwise preference benchmarks, so there is no directly comparable number. As reference points from neighboring settings:
- Self-Critique reports 0.70 average AUC on Qwen2.5-7B-Instruct on RL-MIA (Self-Critique).
- LaRA combined with Self-Critique reports 0.79 ROC-AUC and 31–38% true positives at 5% false positives on RL-MIA Math (LaRA).
We treat item-level AUC above 0.79 with true positives above 38% at 5% false positives, on the full-triple mode, as a meaningful bar. Beating PaCoST-on-margins by a clear, significant AUC gap across seeds is the result that matters.
Two ablations isolate the key ingredients. Removing control subtraction tests whether calibration against style shift is necessary. Collapsing the side split into $D_i$ alone tests whether side information improves detection or only diagnosis. The rephrased-triple mode stresses every method.
Why not just…?
Why not just run n-gram decontamination on the training data? Released reward models rarely come with their preference data, so there is usually nothing to match against. Even when data is available, rephrased contamination evades n-gram overlap: rephrased MMLU lifted Llama-2-13B from about 55 to about 86 undetected (Rethinking Benchmark and Contamination with Rephrased Samples). Our rephrased-triple mode checks whether the score-based audit still catches this case.
Why not just apply PaCoST to the margin? This is the real threat. If side decomposition and control subtraction add nothing, our contribution shrinks to "PaCoST works on reward models," which is still worth reporting but is not the claim. We expect two gains. Control subtraction should matter because rewriting shifts length and style, which reward models reward. Side residuals should detect chosen-only exposure, which barely moves the margin when the rejected side is untouched. The head-to-head comparison with shared paraphrasers settles this. We commit to reporting it as-is.
Why not just use fresh prompts, as RewardBench 2 does? Freshness is a fine defense for new benchmarks but cannot audit models already trained on the old ones. It also expires once the prompts are crawled. The RewardBench 2 gap of about 20 points confounds harder items with leakage, so it cannot tell whether a particular model was contaminated. Our application study tests whether the audit's flags predict which leaderboard models drop most from RewardBench to RB2.
Why not just use likelihood detectors? A reward model's scoring head has no token likelihood. After reward-driven training, likelihood methods fall to near chance (0.42–0.53 AUC) in Self-Critique. We still run Min-K%++ on the base model, because it may win in the chosen-only mode. If it does, that is informative and we will say so.
Proposed Experiments and Decisive Checks
The primary experiment is item-level AUC, and true-positive rate at 5% false positives, for every detector. It is crossed with exposure mode, injection fraction, and model size, with three training seeds.
The first decisive check is the exposure diagnosis. On chosen-only items, $e^-$ should be indistinguishable from controls while $e^+$ is shifted. On full-triple items, both should shift. If the classifier cannot separate these modes above chance, the side-specific claim fails.
The second check targets style confounding. We train a clean reward model that is related in lineage to the generator of the benchmark responses, following the relatedness categories in Preference Leakage. The audit must not flag its benchmark items, because a style preference is not memorization.
The third check is power calibration. Using the fitted $\rho$ and the $\alpha\rho\sqrt{m}$ scaling from When Is Benchmark Contamination Detectable?, we predict the minimum detectable $\alpha$ at the RB2 size of 1,865 items. We then verify that the prediction matches empirical detection rates in the testbed.
Finally, on public RewardBench models, we report which models the dataset-level test flags. We also test whether flag strength correlates with each model's RewardBench-to-RB2 drop, as an external but non-definitive check on real-world validity.
References
- Multi-task bioassay pre-training for protein-ligand binding affinity prediction
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- RewardBench: Evaluating Reward Models for Language Modeling
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
- Detecting RLVR Training Data via Structural Convergence of Reasoning
- LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- RewardBench 2: Advancing Reward Model Evaluation
- What Does Preference Learning Recover from Pairwise Comparison Data?
- DCR: Quantifying Data Contamination in LLMs Evaluation
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models