A model that has memorized a benchmark item has stored the answer, not the premises that determine it. This proposal turns that asymmetry into a contamination test. For each item, we build formally verified "neighbor" versions of the same problem. In some neighbors one premise is flipped, so the correct answer flips. In others a premise is removed, so the correct answer becomes "Unknown." In the rest the surface form changes but the logic does not. A model that reasons will follow the premises. A model that memorized the item keeps giving the original answer where the premises no longer support it. We call that behavioral signature answer stickiness. It is measured from final answers, not token probabilities, so it should survive the post-training that hides contamination from today's detectors.
The need is concrete. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows why current detectors break. These are membership-inference detectors, which ask whether an item has suspiciously low loss under the model; examples are LiRA and Min-K%. LiRA reaches 89.1% AUROC (area under the ROC curve) on a model contaminated by supervised fine-tuning, meaning training on example answers. One epoch of reinforcement learning on clean data then cuts LiRA's AUROC by about 14%. Contamination through chain-of-thought fine-tuning leaves all ten tested detectors at 50–58% AUROC. In both cases the inflated scores remain. Detection has lost its signal while the benefit of cheating persists. Our bet is that the memorized answer, which is exactly what inflates the score, is the one thing post-training cannot hide.
Core Claim
Possible-worlds semantics gives the mechanism. In this view from knowledge representation, what an agent knows is the set of candidate realities ("worlds") consistent with its evidence. A proposition is known only if it holds in every such world.
Three edit types follow from this view. The Interaction with Incomplete Knowledge Bases: A Formal Treatment introduces the language KL, which answers queries with three values: true, false, and unknown. It also shows that what follows from "knowing only α" changes when α changes. Deleting a premise therefore enlarges the set of consistent worlds, and it can turn a determined answer into "Unknown." The Frame Problem and Knowledge-Producing Actions formalizes observations as filters that discard inconsistent worlds. Flipping one observed fact therefore selects a different surviving set, and possibly a different answer. Reasoning About Knowledge and Action uses rigid designators, meaning names that pick out the same object in every world. Consistently renaming them therefore changes nothing logically.
These give three edit types with ground truth computed by a solver: flip, withdraw, and invariant. Our claim is that a contaminated model, even after reinforcement-learning cloaking, returns the memorized answer on flip and withdraw neighbors at a rate well above a matched clean model. This stickiness separates seen from unseen items better than loss-based detectors after cloaking.
We measure stickiness per item. Let $a_0$ be the original gold answer and $V_\neq$ be that item's verified neighbors whose gold answer differs from $a_0$. Then
$$S = \Pr_{v \in V_\neq}[\hat{y}(v) = a_0] - \Pr_{v \in V_\neq}[\hat{y}(v) = a_0 \mid \text{clean reference}],$$
where $\hat{y}(v)$ is the model's answer on neighbor $v$. The second term is the same rate for a reference model never trained on the item. The subtraction removes any general bias toward particular answers. Accuracy on invariant neighbors is reported alongside $S$ as a control: low stickiness is only meaningful if the model has not collapsed on all variants.
Relation to Prior Work
Two papers motivate the gap. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that loss-based detection fails under reinforcement learning and chain-of-thought contamination; we replace the loss signal with a behavioral one. The Emperor's New Clothes in Benchmarking? tests 20 benchmark rewrites. It finds that surface rewrites preserve fidelity (above 0.9) but give almost zero contamination resistance, because contaminated models map small variations back to the memorized answer. Aggressive rewrites drop MMLU fidelity to 0.686.
We read that negative result as a positive signal. For mitigation, it is bad that memorized answers survive surface edits. For detection, it is exactly the fingerprint we want, provided the edit changes the correct answer. That paper's rewrites were never verified to change or preserve the answer, which is precisely what formal semantics adds.
Three papers supply the method and data. MindGames procedurally generates dynamic epistemic logic puzzles, which track how public announcements change what each agent knows, with verifiable answers. It uses them for fresh evaluation, not for detecting whether a given item was seen; we reuse its generator to make neighbors of fixed items. Compositional Consistency-Guided Decoding enforces consistency between a hypothesis and its negation on FOLIO's first-order logic annotations. It documents epistemic abstention, where models answer "Unknown" from their own uncertainty rather than because the premises underdetermine the answer. It never asks whether its gains differ on contaminated items. Our withdraw edits are the controlled version of its Unknown class. Can Large Language Models Reason? A Characterization via 3-SAT shows that models answer satisfiability problems from shortcuts such as prompt length and that MiniSAT verifies answers exactly. It supplies our cheapest fully verified domain. It also warns that length must be matched across neighbors, or stickiness will partly measure a length heuristic.
Research Program
We would build a neighbor generator for three formally annotated sources: FOLIO's first-order logic fields, the MindGames setups, and 3-SAT instances in both symbolic form and the natural-language "menu" form. For each item, the generator enumerates single-premise edits. These are negating a premise (flip candidate), deleting a premise or announcement or changing who can observe what (withdraw candidate), and renaming entities or reordering premises (invariant). An off-the-shelf first-order prover, the dynamic epistemic logic model checker, or MiniSAT labels each neighbor True, False, or Unknown. Only neighbors whose label is certified are kept.
Edits are then realized in the item's own wording by template substitution, not free rewriting. Neighbors must match the original in length and vocabulary within a tolerance, which blocks the shortcut documented in the 3-SAT study.
Contamination is created under control, following the two stages of the fragility paper. In Stage I, Qwen2.5-7B-Instruct receives supervised fine-tuning on a random half of each item pool, followed by one epoch of clean GRPO, a reinforcement-learning method for reasoning models. In Stage II, DeepSeek-R1-Distill-Llama-8B is fine-tuned on chain-of-thought solutions for half the items. The other half of each pool serves as non-members. We also keep an uncontaminated twin of each model, trained identically on unrelated data, to serve as the clean reference in $S$.
Baselines
The naive reference is the score gap: accuracy on the original item minus accuracy on its invariant neighbors. It needs no formal labels. If it matches stickiness, the formal machinery is unnecessary. The fragility paper and the mitigation study both suggest it will be weak, since memorized answers survive invariant edits by design.
The strongest existing detector is LiRA as run in On The Fragility of Benchmark Contamination Detection in Reasoning Models, along with Min-K% and loss thresholding. LiRA is the strongest because it compares the target model's likelihood to that of reference models and was the best detector before cloaking. That paper reports 89.1% AUROC before reinforcement learning, a drop of about 14% after one GRPO epoch, and 50–58% AUROC averaged across detectors in Stage II. Those numbers come from math benchmarks such as AIME and GPQA Diamond. No reported result was found on Lacuna for any of these detectors on FOLIO, MindGames, or 3-SAT, so we rerun all of them on our member/non-member splits. They get the same reference models, the same model checkpoints, and the same held-out calibration set as stickiness.
Beating LiRA means two things. First, item-level AUROC after cloaking must be at least 75%, with a paired bootstrap confidence interval that excludes LiRA's post-cloaking AUROC. Second, stickiness must lose less than LiRA does between the pre-reinforcement-learning and post-reinforcement-learning checkpoints.
The key ablation isolates formal verification. We replace solver-certified neighbors with neighbors an LLM writes when told to "change a premise so the answer changes." Everything else, including the stickiness statistic and the length matching, stays fixed. We report both detection AUROC and the fraction of LLM-written neighbors whose claimed label a solver refutes. A second ablation drops withdraw edits and keeps only flips. This tests whether the three-valued structure adds signal beyond binary contrast.
Why not just…?
Why not just rewrite the benchmark with an LLM and compare scores? The Emperor's New Clothes in Benchmarking? found that surface rewrites give almost zero contamination resistance and that stronger rewrites destroy fidelity. Score comparisons on rewrites therefore either miss contamination or confound it with a change in difficulty. The score-gap baseline tests this directly on our splits.
Why not just use membership inference and accept some loss in AUROC? The loss after cloaking is not small. The fragility paper reports reference-free detectors falling to about 50% AUROC after one GRPO epoch and all ten detectors at 50–58% in Stage II. We rerun these detectors rather than assume the result transfers to logic tasks.
Why not just write counterfactual contrast items with an LLM, with no logic involved? This is the real threat, since it keeps our core insight and drops the formal apparatus. Our concern is label noise. An LLM-authored "flipped" item whose true answer did not flip will register as stickiness in a clean model and blur the separation. Unknown-class edits are exactly where models are unreliable, as the epistemic-abstention finding in Compositional Consistency-Guided Decoding shows. The verification ablation settles the question. If LLM-written neighbors match certified ones in AUROC and are rarely refuted by the solver, we will report that the formal layer is unnecessary for these domains.
Why not just evaluate on fresh generated items, as DyVal does? DyVal measures capability on new items, and it found that phi-1.5 and WizardMath-13B score near zero despite strong static results. That tells us a model is weak, not which items it saw. Neighbors are anchored to specific benchmark items, which is what an audit of a leaderboard score requires.
Proposed Experiments and Decisive Checks
The primary experiment reports item-level AUROC for stickiness and all baselines. It covers the FOLIO, MindGames, and 3-SAT splits at three checkpoints: after contamination by supervised fine-tuning, after clean GRPO, and after Stage II chain-of-thought contamination.
The decisive test is the difference-in-differences across the reinforcement-learning checkpoint. The fragility paper's analysis says GRPO's importance-ratio clipping shrinks the gap in log-probability between member and non-member items. It says nothing that would remove answer-level recall, and the inflated pass@1 persists. So stickiness should stay roughly flat while LiRA drops. If stickiness drops by as much as LiRA, the hypothesis is wrong.
We also check fidelity in the mitigation study's sense. On uncontaminated twins, accuracy on certified neighbors should match accuracy on freshly generated items of equal proof depth. This ensures neighbors are not simply harder. Invariant-neighbor accuracy guards against trivial detection by general collapse.
Stage II is where we expect partial failure, and we state it up front. If chain-of-thought fine-tuning truly teaches the reasoning schema, members and non-members may become behaviorally indistinguishable item by item. The 11.7% score inflation would then have to be detected at the level of the whole benchmark, as excess stickiness in the population relative to the clean twin. We will report both levels.
Finally, we apply the detector without training labels to public checkpoints on the original FOLIO test set. This is an exploratory audit, not a validation.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- The Interaction with Incomplete Knowledge Bases: A Formal Treatment
- The Frame Problem and Knowledge-Producing Actions
- Reasoning About Knowledge and Action
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
- MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
- Compositional Consistency-Guided Decoding for Three-Way Logical Question Answering
- Can Large Language Models Reason? A Characterization via 3-SAT
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks