When a language model gets a benchmark question right, we usually cannot tell whether it worked the answer out or remembered it. Existing contamination detectors check whether the model is unusually confident on the exact benchmark text. Those detectors break down on modern reasoning models. Reinforcement-learning post-training erases the confidence gap, and models trained on leaked worked solutions generalize to paraphrases. This proposal uses a different kind of evidence. In pragmatic reference games, the correct answer is logically tied to other judgments the model can be asked about separately, so we check whether the model's correct answer is actually supported by its own reasoning about the same situation. A model that reasons will usually be right because its piecewise judgments add up. A model that memorized the benchmark will often be right in spite of them.
The pieces come from the Rational Speech Acts model (RSA). RSA is a standard account of how listeners recover intended meaning by asking what a cooperative speaker would have said instead. Suppose a speaker says "blue" in a scene with a blue square, a blue circle and a red square. A pragmatic listener concludes the speaker meant the square, because a speaker who meant the circle would have said "circle." That answer is a Bayesian inversion of a speaker model, and the speaker model rests on literal truth judgments. Each level can be queried directly from a language model. We call the proposed signal unsupported correctness: a direct answer that is right while the answer rebuilt from the model's own speaker and literal judgments is wrong. We claim this signal occurs more often on contaminated items than on clean ones, and that it survives the training regimes that defeat likelihood-based detectors.
Core Claim
Leaked benchmark data trains a model on the question the benchmark asks. Here that is the listener question: "which object did the speaker mean?" Leaked data rarely trains the counterfactual questions about the same scene, such as "if you wanted to pick out the blue circle, what would you say?" or "is 'square' literally true of the red square?" The RSA structure makes these counterfactual questions mathematically bound to the benchmark answer. The mismatch between the direct answer and the answer implied by the model's own lower-level judgments is therefore a within-item test.
This test needs no clean reference model, no difficulty calibration across item sets, and no token log-probabilities. It only needs repeated sampled answers. Those are exactly the three constraints the file identifies as limiting current detectors (PaCoST, ConStat).
Concretely, a benchmark item gives us a scene $C$ of candidate objects, an utterance $u$ and a target $o^*$. We elicit three things from the model. The first is its direct listener distribution $L^{dir}(o \mid u, C)$. The second is its speaker distribution $S_M(u' \mid o, C)$ over a fixed set of alternative utterances $U(C)$ for every object $o$ in the scene. The third is its literal truth judgments for every utterance–object pair. The recomposed listener is the one-step Bayesian inversion of the model's own speaker:
$$L^{rec}(o \mid u, C) = \frac{S_M(u \mid o, C)\, p(o)}{\sum_{o'} S_M(u \mid o', C)\, p(o')},$$
where $p(o)$ is uniform over the scene. A second recomposition runs full RSA upward from the literal judgments, with a single rationality constant fitted on held-out clean items. The per-item score is
$$\Delta(C, u) = L^{dir}(o^* \mid u, C) - L^{rec}(o^* \mid u, C).$$
This is the probability of being right directly minus the probability of being right from one's own reasoning. A suspect item set is flagged when its mean $\Delta$ exceeds that of a matched control set, using a paired test in the spirit of PaCoST.
The score is directional on purpose. Models are known to be globally inconsistent across pragmatic levels. Are LLMs good pragmatic speakers? finds only weak correlation between Llama-3-8B-Instruct's speaker scores and RSA predictions on TUNA. FANToM shows GPT-4 above 70% on multiple-choice belief questions but at 12.3% when all related formats must agree. That baseline inconsistency would swamp an undirected disagreement score. Our bet is that memorization produces a specific asymmetric inconsistency: correct at the top level, unsupported below it.
Relation to Prior Work
Two lines of work motivate the proposal. The first shows that current detectors fail on reasoning models. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates models through fine-tuning, then applies reinforcement learning (GRPO). The RL step shrinks the likelihood gap, and all ten detectors fall to 50–58% AUROC when contamination arrives as worked solution traces. Detecting Data Contamination from Reinforcement Learning Post-training responds with Self-Critique, which asks the model for an alternative solution and measures whether its output entropy has collapsed onto one path. Both papers study only math and logic. Our signal differs from both: it asks whether the model's reasoning about other questions in the same scene supports the answer, not whether the benchmark text or solution path looks familiar.
The second motivating line shows that reframing questions separates memory from reasoning. Beyond Memorization finds that turning cloze questions into multi-step reasoning questions drops influence-function source retrieval from 77.5% to 17.5% and removes post-cutoff decay. We borrow the intuition but use it for detection rather than mitigation.
Several papers use pragmatic or Theory-of-Mind structure to avoid contamination rather than to detect it. DECRYPTO builds an interactive signaling game. ExploreToM searches a story grammar for fresh adversarial items. FANToM uses multi-format consistency to expose "illusory" belief tracking. None of them turns RSA's inversion into a statistic for auditing an existing benchmark. FANToM's "All" score is the nearest relative, and we treat it as an ablation.
The method draws on RSA itself, as surveyed in Pragmatics in Language Grounding and made neural in Colors in Context. One constraint shapes the design: inverting open-ended generation is intractable, so the file's open problems note that restricted candidate sets are needed. We therefore work only with benchmarks whose alternative sets are closed or can be enumerated. These are reference games (TUNA furniture, Colors in Context) and scalar-implicature items (IMPPRES), where the alternatives are the lexical scale ("some"/"all").
Research Program
The work has three parts.
The first part builds the probe suite. For each benchmark family we write fixed prompt templates for the three levels and define the alternative set $U(C)$. In reference games this is the attribute words and their pairs. In IMPPRES it is the scale mates. We estimate each distribution from $k$ sampled answers per prompt, so the method runs on APIs without log-probabilities. Our own elicited distributions are noisy, so $\Delta$ is reported with bootstrap intervals. We also measure how many samples are needed for a stable item score.
The second part is a controlled contamination study, which gives us ground truth. We start from Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct. We contaminate a random half of each benchmark's test items under four regimes that mirror the Fragility and RL-MIA setups:
- verbatim fine-tuning on question–answer pairs;
- fine-tuning on worked solution traces written by a stronger model;
- solution-trace fine-tuning followed by GRPO on the same items, the "cloaking" condition;
- GRPO-only exposure.
A fifth condition is essential. We fine-tune the base model on freshly generated reference games, drawn from the same scene grammar but with no benchmark items. This produces genuine skill gain without contamination. A useful detector must stay silent here even though benchmark accuracy rises. That is what separates "learned the benchmark" from "learned pragmatics."
The third part is an exploratory audit of public open-weight models on the existing TUNA, Colors and IMPPRES items. It compares their $\Delta$ against procedurally generated isomorphic scenes that we can guarantee are unseen. There is no ground truth here, so this part is reported as evidence, not validation.
Baselines
The naive reference point is the raw accuracy gap between original benchmark items and freshly generated isomorphic items. This is what an auditor would do without any detector. It is confounded by difficulty differences between the two sets, and we run it on the same items and models as everything else.
The likelihood family (loss, Min-K% probability, perplexity and LiRA) is run with log-probability access on the open models. It represents the standard practice. The file reports these at 0.42–0.53 AUC on RL-MIA (Self-Critique paper). It also reports that LiRA reaches 89.1% AUROC before RL but that detectors fall to roughly 50% after GRPO and to 50–58% after solution-trace contamination (Fragility).
PaCoST compares confidence on original versus paraphrased questions. ConStat compares performance on the benchmark against rephrased variants, calibrated with reference models. ConStat reports detecting 98.4% of sample-specific contamination against 71% for the best prior baseline in its controlled setting. We run both with the same rephrasings and give ConStat the uncontaminated base checkpoint as its reference model. That is a more favorable reference than it would have in practice.
The strongest competitor for the hard regimes is Self-Critique. It is the only method in the file designed for RL-stage contamination, reporting an average AUC of 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct on RL-MIA. There is no reported result found on Lacuna for any of these detectors on pragmatic or reference-game benchmarks.
Fairness is enforced by a matched query budget. Our probe needs roughly one query per object–alternative pair per sample. Every baseline is given the same total number of model calls, spent on more paraphrases or more critique samples, and the same fixed seeds and prompt-tuning budget on a shared held-out development split.
Two ablations isolate the key ingredient. The first replaces directional $\Delta$ with an undirected disagreement rate between $L^{dir}$ and $L^{rec}$, a FANToM-style consistency score. If this matches $\Delta$, then the RSA direction is not doing the work. The second replaces recomposition with the answer-flip test described below.
Beating the baselines means the following. In the solution-trace and cloaking conditions, item-level AUROC must be significantly above Self-Critique run on the same models and items, by non-overlapping bootstrap intervals, and above the 50–58% range reported for likelihood detectors. In addition, the false-positive rate in the genuine-skill condition must stay at the nominal level.
Why not just…?
Why not just use paraphrase-based detection like PaCoST or ConStat? These methods detect memorized wording. The Fragility paper shows that solution-trace contamination teaches a procedure that transfers to paraphrases, which drives detectors toward chance. Our cloaking condition reproduces that setting, so the comparison is direct. If PaCoST or ConStat stays strong on pragmatic items even there, the result is still informative: it would mean pragmatic benchmarks are less susceptible to procedural generalization than math. We would report that result rather than our method.
Why not just flip the scene so the pragmatic answer changes, and check whether the model follows? This is the most serious threat, because it is simpler and targets the same weakness. An example is adding a red circle so that "blue" now picks out the circle. We take it seriously in two ways. First, flipped items carry their own difficulty, so a clean model also fails some of them. Without reference-model calibration the test inherits ConStat's dependence on clean references, whereas $\Delta$ is computed within one scene. Second, a model trained on worked traces for the original scene may handle one flip correctly while still lacking a coherent speaker model. Neither point settles the question in the abstract. The flip test is therefore run as a first-class ablation at matched query budget. If it matches $\Delta$ in every regime, we will recommend it and present recomposition only as a diagnostic of why items are flagged.
Why not just use Self-Critique, which already targets RL contamination? Self-Critique measures whether the model can produce a different solution path. Reference-game answers are short, so there is little path over which entropy can collapse, and it is unclear the signal transfers. It is our strongest baseline, and the two signals may be complementary. We report their combination as well as each alone.
Why not just report a consistency score like FANToM's "All"? Clean models are already badly inconsistent across pragmatic levels, per the TUNA and FANToM results cited above. An undirected score would therefore flag every model. The directional ablation tests whether "right on top, unsupported below" is the specific signature of memorization.
Proposed Experiments and Decisive Checks
The main experiment crosses the two base models, three benchmark families and five training conditions. It reports item-level AUROC, benchmark-level detection power at a 5% false-positive rate, and how well mean $\Delta$ tracks the true accuracy inflation that each contamination run produces.
Three outcomes would be decisive.
The first is survival under cloaking. In the solution-trace-plus-GRPO condition, $\Delta$ should remain well above chance while likelihood detectors reproduce the near-chance collapse reported by Fragility. If $\Delta$ also collapses, the core claim is false.
The second is silence under genuine learning. In the fresh-games condition, benchmark accuracy should rise without $\Delta$ rising. If $\Delta$ fires here, it measures skill change rather than contamination, and the method fails.
The third is an adversarial stress test. We add a sixth contamination condition whose leaked traces explicitly narrate RSA counterfactuals ("had the speaker meant the circle, they would have said 'circle'"). Such traces may teach the speaker questions too and close the gap. We predict a weaker but non-zero signal, because the traces cover only the scene's intended target while the speaker probes query every object. If the signal vanishes, recomposition is only robust to contamination that lacks explicit pragmatic reasoning, and we will state that limit plainly.
A secondary check tests whether prompt templates, rather than contamination, drive $\Delta$. We rerun the clean models with three template families and require control-set $\Delta$ to be stable across them before any audit result is reported.
References
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Are LLMs good pragmatic speakers?
- FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
- Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
- The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind
- Explore Theory of Mind: program-guided adversarial data generation for theory of mind reasoning
- Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
- Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding
- Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition