Factorial Pragmatic Signatures as Contamination-Robust Evaluation

When a language model reads "some of the cookies are gone" and concludes "but not all," two very different things could be happening. The model may be reasoning about what the speaker knew and what else the speaker could have said. Or it may be replaying a pattern it has seen millions of times on the web: "some" usually means "not all." Every current benchmark of this kind of inference, called a scalar implicature, scores models with accuracy on fixed text items. Fixed items can leak into training data. More damagingly, a plain default heuristic already produces the "right" answer on most of them. Leakage detectors that work by checking how familiar the text looks to the model stop working once models are post-trained with reinforcement learning. As a result, nobody can currently tell whether a model's pragmatic competence is real.

We propose to score pragmatic reasoning with a signature rather than with accuracy. The signature comes from cognitive science. Humans draw the "not all" inference less when the speaker saw only part of the scene. When numbers ("one," "two," "three") have recently been mentioned, this drop partly reverses: explicit numerals make the stronger alternatives salient, so the listener infers "not all" more even from a partially informed speaker. This is an interaction between two factors, speaker knowledge and the presence of numeral alternatives. A model that has memorized "some → not all," or that has memorized answer labels, can match the average rate of the inference without reproducing this crossover. We will build a procedural generator for fully crossed scenarios and score models on the interaction contrast measured on fresh items. We will then deliberately contaminate models to test whether this score, unlike accuracy and unlike current leakage detectors, separates template memorization from real competence.

Core Claim

For pragmatic inference, the right contamination-robust quantity is a theory-predicted interaction between factors, estimated on procedurally generated items. It is not accuracy, even on fresh items. Define a model's endorsement of the "some but not all" reading, on a logit scale, in each cell of a 2×2 design that crosses speaker knowledge $K\in\{F,P\}$ (full or partial) with numeral context $A\in\{N,0\}$ (numerals present or absent). Call this $\ell_{K,A}$. The signature is

$$I = (\ell_{P,N}-\ell_{P,0}) - (\ell_{F,N}-\ell_{F,0}),$$

which human data and the Rational Speech Act model predict to be positive. The Rational Speech Act (RSA) model treats a listener as a Bayesian reasoner about a speaker who chooses utterances to be informative. Taking differences of differences cancels any additive bias toward the pragmatic reading, so a model that over-applies implicatures does not earn credit. The logit scale limits ceiling artifacts. Memorizing surface templates moves the main effects, not $I$.

The human anchor is Scalar Implicature is Sensitive to Contextual Alternatives. Across four preregistered experiments, it found a consistent knowledge effect and a knowledge × numerals interaction, significant in Experiments 2 and 4 ($p=.018$, $p=.017$), a trend in Experiment 1 ($p=.0935$), and absent in Experiment 3, where material similarity made numerals act as a main effect. That fragility is the main risk to this proposal, and the program below treats it directly.

How This Differs From Existing Work

Three lines of work motivate the gap. Static pragmatic probes test fixed sentences or scales. Pragmatic inference of scalar implicature by LLMs uses 198 British National Corpus sentences. Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics finds that models rank adjective intensity well but fail at human-like scalar diversity, with GPT-4 conservative and Flan-T5-xxl ahead of it. PaCE builds context-flip pairs and reports an average Context Sensitivity Gap of −14.0% against a human +1.5%. All three use fixed items. Only PaCE scores a contrast, and it is a single-factor flip. Our signature needs two factors to combine, and our items are generated fresh.

Dynamic evaluation supplies the method. DyVal generates reasoning problems from graphs on the fly and exposes models that score near zero off their static benchmarks. It explicitly leaves pragmatics open, because pragmatics lacks verifiable labels. Our answer to that problem is to verify a predicted pattern rather than individual labels.

Contamination detection constrains the design. ConStat defines contamination as performance that does not generalize to a reference set. Data Contamination Calibration for Black-box LLMs perturbs text to detect membership. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that clean reinforcement learning drives detectors toward chance. All three treat contamination as memorized text or memorized answers. We target a third kind: memorized interpretive defaults, which fresh text alone does not remove.

Research Program

The generator. It produces vignettes in which a speaker observes $k$ of $n$ objects, with $n$ from 3 to 10 and $k$ graded rather than binary. This yields a dose-response curve instead of a single full/partial contrast. The speaker then utters "some." Numeral context is manipulated by whether preceding discourse, such as other characters' reports, uses numerals. Distractor numerals unrelated to the scene serve as a control for mere token priming. Surface features vary independently of the design cells: characters, object types, sentence templates, and the language model used to paraphrase. Varying these guards against the Experiment 3 failure mode.

Predicted patterns. Each item receives a quantitative prediction from an RSA model in the style of Zhang et al. The prior over world states is varied following The Role of Prior Beliefs in The Rational Speech Act Model of Pragmatics, since strong priors can produce anti-exhaustive readings and so offer a second, independent signature.

Human grounding. Because the human interaction is small and sometimes absent, we will run a preregistered human study on a stratified sample of generated items, about 200 items with enough raters per cell to estimate $I$ with a confidence interval. The benchmark's ground truth is the human-estimated $I$ and dose-response slope. RSA fits serve as a check.

A mechanism check. This borrows from Predicting scalar diversity with context-driven uncertainty over alternatives, which found that entropy over the alternatives a model predicts tracks human implicature rates ($p=0.001$) while surprisal of the strong scalemate does not ($p=0.928$). For open models, we will measure whether numeral context reduces the model's entropy over the alternative expressions a speaker could have used, and whether that reduction statistically mediates $I$. A model that shows $I>0$ without the entropy shift is suspect.

The contamination testbed. Using open models, including DeepSeek-R1-Distill-Llama-8B from the fragility study, we will fine-tune under three conditions. Condition (a) uses Zhang et al.'s published items with human-majority labels. Condition (b) uses static pragmatic items, such as the British National Corpus scalar items. Condition (c) uses a large "some implies not all" template corpus. A fourth, competence condition fine-tunes on RSA-generated data with full factorial variation, disjoint in surface form from the evaluation. Each condition is followed or not by one epoch of clean GRPO, a reinforcement learning method, reproducing the cloaking setup of the fragility paper.

Baselines

The naive reference point is accuracy on static items: the Zhang et al. materials and the British National Corpus items. This is what current papers report, and we expect it to rise under every contamination condition.

The strongest detector-style competitor is the membership-inference family, run through the fragility paper's protocol: LiRA, Min-K%, and loss, plus PAC from the black-box calibration paper. All are applied to the same checkpoints with identical data access. The fragility paper reports LiRA at 89.1% AUROC after contamination by supervised fine-tuning, falling to about 50% after clean GRPO, and all ten detectors at 50–58% AUROC in its Stage II. We expect to replicate this collapse on pragmatic items. Beating it means our signature separates contaminated-only from competence-trained checkpoints above 0.8 AUROC after GRPO, where these detectors sit near chance.

The strongest performance-based competitor is ConStat. It reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models. We will run it with our generated items as its reference benchmark and with its original fleet-calibration procedure, using the same pool of reference models. ConStat asks whether static accuracy exceeds fresh accuracy. The distinguishing test is condition (c): template contamination raises accuracy on both static and fresh items, so ConStat should see no gap, while $I$ should stay flat. No reported result on Lacuna exists for ConStat on pragmatic benchmarks.

PaCE's Context Sensitivity Gap is computed on our items by adding literal-context versions. It is a conditional metric and therefore the closest competing contrast.

The key ablation scores the same generated items two ways, as accuracy and as $I$. This isolates whether the benefit comes from freshness, which both scores share, or from the interaction structure, which only $I$ uses. No reported result was found on Lacuna for any language model on the knowledge × numerals interaction.

Why not just…?

Why not just generate fresh items and report accuracy, DyVal-style? This is the real threat. If fresh-item accuracy already separates contaminated from competent checkpoints, the interaction adds nothing, and we will say so. Our reason to expect otherwise is that the dominant contamination here is an interpretive default rather than specific strings. PaCE's finding that models systematically over-apply pragmatic readings is consistent with such a default. A default of this kind carries over perfectly to fresh items. The ablation settles the question: under condition (c), fresh accuracy should rise while $I$ does not.

Why not just run membership-inference detectors on the static benchmarks? The fragility paper shows that clean reinforcement learning drives them to chance. These detectors also require log-probabilities that black-box APIs often withhold, as the PAC paper notes. Our signature needs only sampled ratings. We will nonetheless run these detectors on every checkpoint rather than assume they fail.

Why not just use PaCE's context-flip gap? A single flip can be satisfied, or broken, by instruction-following alone. PaCE itself reports GPT-4o's gap swinging from −10.4% to +26.1% under strict prompting, a change driven by surface instructions rather than by reasoning about the speaker. The two-factor interaction cannot be produced by a single literal/pragmatic switch. The experiment reports both metrics side by side on the same checkpoints.

Why not just prompt with chain-of-thought? That is an intervention, not an evaluation. The contamination question remains. We will include chain-of-thought as a scoring condition to check that $I$ is stable across prompting formats.

Decisive Checks

The proposal succeeds only if the following hold together. First, the human study reproduces $I>0$ on generated items with a confidence interval excluding zero. If it does not, the signature has no valid anchor, and we fall back to the graded-knowledge dose-response, which rests on the robust knowledge main effect. Second, conditions (a) through (c) inflate static accuracy, and the detectors lose their signal after GRPO. Third, $I$ on fresh items does not rise under conditions (b) and (c) but does rise under the competence condition, which the accuracy ablation fails to show. Fourth, the distractor-numeral control leaves $I$ unchanged, ruling out priming by numeral tokens. Condition (a) is the interesting edge case. If memorized Zhang items produce $I>0$ on fresh surface forms, that is transfer of the pattern, which we would count as learned competence and report as such. It would not be a failure of the metric.