When a language model has seen a benchmark's test set during training, it can return the right answer without doing the reasoning. Current detectors for this kind of leakage either look for overlapping text in the training corpus or look for unusually confident token probabilities. Both now fail in common settings. Paraphrased leaks slip past text matching. A short round of reinforcement learning on clean data erases the probability signal. We propose a test that looks only at the model's answers. Take a public logic benchmark item and change exactly one premise, so that the correct answer provably flips while the rest of the text stays word-for-word identical. A model that reasons will follow the flip. A model that memorized the item will tend to give the old answer anyway, and we call that answer "sticky." Then compare the model's stickiness on edited published items against its stickiness on freshly generated items with the same logical structure and the same edit. That comparison separates memorization from ordinary difficulty.
What makes this possible is modular logic translation. A translation is modular when each sentence or rule maps to logic on its own, so adding or changing one rule changes only that rule's translation. Benchmarks like FOLIO already pair each premise sentence with its own first-order formula. That pairing lets us edit one sentence, verify the edited sentence against its formula, and recompute the gold label with a theorem prover, without paraphrasing anything else. The surface text that triggers recall stays intact, and the label is guaranteed by a solver rather than an LLM judge.
Core Claim
A behavioral probe built from single-premise, solver-verified edits gives a contamination signal that survives the two evasions that break current detectors: paraphrased training data, and reinforcement learning applied after contaminated fine-tuning. The probe needs only black-box access to the model.
Define the probe as follows. For a model $M$, let $S_{\text{pub}}$ be the rate at which $M$ outputs the original gold label on edited published items whose gold label has flipped. Let $S_{\text{fresh}}$ be the same rate on never-published isomorphs, meaning the same formulas with renamed entities and predicates, re-verbalized and given the identical edit. The memorization gap is
$$\Delta = S_{\text{pub}} - S_{\text{fresh}}.$$
Under a clean model, $\Delta$ should be near zero. Edit difficulty and reasoning shortcuts act on both arms equally, because the arms are logically identical. Under contamination, $\Delta$ should be positive. For a per-item score, we use the count of sticky answers across $k$ independent flip-edits of the same item, minus the matched count on its isomorph.
Three neighbouring works shape this claim. A²RBench remaps symbols, for example swapping 0 for a, and reports a 17.7% accuracy drop. That changes the surface while keeping the logic. We do the reverse: we keep the surface and change the logic. DyVal avoids contamination by generating fresh reasoning graphs, but it cannot say whether a score on an existing benchmark is inflated. Can Large Language Models Reason? A Characterization via 3-SAT uses synthetic formulas to sidestep memorization and translates whole problems to a solver in one pass. None of the three edits a published item locally or uses the edit as a measuring instrument.
Research Program
Classical edits on FOLIO. The first component is an edit engine for FOLIO. FOLIO contains 1,435 stories with expert first-order annotations for each sentence, and its gold labels are checkable with Prover9 or Z3. For each item, we search over single-formula edits: deleting a premise, negating it, swapping a quantifier, or adding one new premise. We keep only edits for which the prover shows that the conclusion's label changes among true, false, and unknown.
The edited formula is turned into English one sentence at a time by an LLM. We accept the sentence only if a separate model translates it back into a formula that the prover shows is equivalent to the target formula. This is the cycle-consistency idea from A²RBench, applied to declarative logic rather than Python. All other sentences stay byte-identical.
Fresh isomorphs are built by renaming every predicate and constant in the annotated formulas and re-verbalizing them sentence by sentence through the same round-trip check. They are generated at evaluation time and never released.
Nonmonotonic edits. The second component extends the probe to reasoning where adding a premise can retract a conclusion. Classical first-order logic cannot do this: adding a premise never removes an entailment, so in FOLIO a flip from true to unknown requires deleting a premise. Default-style reasoning is different. We add "typically X; Y's are exceptions" premises and compile them into Answer Set Programming, a rule language where conclusions can be withdrawn when new rules arrive. The compilation uses abnormality atoms, which mark a case as an exception.
Rule addition stays syntactically modular. The semantic effect can be global, and the classical results explain why. Results on Translating Defaults to Circumscription and The Power of Beliefs prove that defaults with prerequisites have no faithful modular translation into circumscription or autoepistemic logic. In those formalisms, a one-rule edit cannot be accounted for by changing one rule. We use that property: an edit that adds one sentence but forces the solver to recompute the whole theory should be especially hard to answer from a memorized label. For causal-style and weighted premises, the sound modular compilers of A Logic Program Characterization of Causal Theories and On the Relationship between P-log and LPMLN supply translations into answer-set programs with proven correctness. We reuse them; we do not prompt an LLM to invent encodings.
Controlled contamination. The third component follows the protocol of On The Fragility of Benchmark Contamination Detection in Reasoning Models, which knows exactly which items were leaked. We fine-tune Qwen2.5-7B-Instruct on a random half of FOLIO test items under three conditions. The first uses verbatim items. The second uses rephrased items in the style of Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. The third uses items with chain-of-thought solutions. Each contaminated model is then given one epoch of GRPO, a policy-gradient reinforcement learning method, on clean logic data. Since the leaked items are known, every detector can be scored by AUROC at the item level.
Baselines
Naive reference. Raw accuracy on the original items and on fresh isomorphs, with no edits. A gap between these two arms already hints at contamination. It cannot tell memorization apart from the isomorphs simply being worded differently, and it gives no per-item evidence. Our probe must beat this gap's AUROC on the same leaked/clean split.
Strongest loss-based detector. LiRA, run with the Fragility paper's settings, along with Min-K% and plain loss. That paper reports LiRA at 89.1% AUROC after contaminated fine-tuning, falling by about 14% after one GRPO epoch. Loss falls by about 16%. Chain-of-thought contamination drives all ten detectors to 50–58% AUROC. These are the strongest existing methods on the question we target, and they use the same model access we use plus log-probabilities. We run them on the same checkpoints and the same item split. Beating them means matching LiRA within a few points before reinforcement learning, and staying clearly above its post-reinforcement-learning AUROC afterwards. No reported result found on Lacuna for any detector on FOLIO specifically.
Corpus-side detector. The LLM Decontaminator of Yang et al., which reports F1 of 0.94–1.0 on rephrased samples. It needs the training corpus, which we have in the controlled setting and rarely have for deployed models. We run it with that corpus access as an upper reference for what full data access buys.
Surface-edit probe. A²RBench-style symbolic remapping applied to the same FOLIO items: rename entities and predicates while keeping the logic and label fixed, and measure the accuracy drop. It is the closest behavioral competitor.
Monolithic-edit ablation. This isolates our key ingredient. An LLM rewrites the whole story with the changed premise in one pass, so the surface text is no longer verbatim, and the label is recomputed by the solver. If this ablation matches our probe, keeping the surface fixed does not matter and modularity is doing no work.
Non-flipping control. Edits that change one premise without changing the label. These measure plain sensitivity to perturbation.
For context on difficulty, FOLIO reports GPT-4 few-shot reasoning at around 70% and neuro-symbolic Logic-LM/DetermLR at around 77–78%.
Why not just…?
Why not just benchmark on freshly generated problems, as DyVal does? Fresh generation sidesteps contamination but does not measure it. The field keeps reporting FOLIO, MMLU, and similar static suites, and the question is whether those numbers are inflated. A fresh-versus-published accuracy gap also mixes memorization with differences in wording and difficulty. Our fresh isomorphs are used as the control arm of a matched comparison, not as a replacement benchmark.
Why not just remap symbols, as A²RBench does? This is a real threat. Remapping is cheaper and needs no prover. Our concern is that rephrased contamination, which raised a 13B model from about 55 to about 86 on MMLU in Yang et al., teaches models answers that survive surface change. A remap-robust memorizer would then look clean. A label-flip edit attacks the memorized answer directly. The rephrased-contamination arm settles the question: if remapping AUROC matches ours there, we will report that the extra machinery is unnecessary.
Why not just run a corpus decontaminator or n-gram filter? Yang et al. show that n-gram overlap misses paraphrased leaks. Their stronger LLM judge needs the training data, which is unavailable for most deployed models. Our probe uses only black-box answers. In the controlled setting we report both, so the cost of losing corpus access is visible.
Why not let an LLM rewrite and relabel edited items? The labels would be unreliable. FOLIO reports GPT-4's natural-language-to-logic translation at 65.21% execution accuracy, so LLM-relabeled flips would often be wrong. Whole-story rewriting also discards the verbatim text whose presence is what exposes memorization. The monolithic-edit ablation tests the second point directly.
Proposed Experiments
Calibration. On an uncontaminated base model, $\Delta$ should be statistically indistinguishable from zero. If it is not, the isomorph construction is biased, and we fix the construction before running anything else.
Robustness to evasion. Across the verbatim, rephrased, and chain-of-thought contamination arms, we report per-item AUROC before and after GRPO for our probe and every baseline. The central prediction is that our AUROC changes little after reinforcement learning while LiRA's falls toward the reported roughly 50% floor. The rationale is that GRPO compresses probability gaps but has no particular reason to dislodge a memorized label. The claim is falsified if our probe collapses after GRPO too.
Does modularity matter? We compare the monolithic-edit ablation and the non-flipping control against the full probe. The full probe should beat both if verbatim context plus a verified flip is what matters.
Nonmonotonic edits. We compare stickiness under retraction edits, the exception premises compiled to answer-set programs, with stickiness under classical edits. We predict a larger $\Delta$ for retraction edits, because a memorized "true" must be actively withdrawn rather than left unchanged.
Second benchmark. We repeat the controlled pipeline on the SAT-Menu problems of Hazra et al., using single-clause edits checked by MiniSAT. This tests whether the effect holds outside first-order stories.
Exploratory audit. Finally, we report $\Delta$ on FOLIO for several open-weight models whose training data is unknown. We present these as evidence, not verdicts, because there is no ground truth for those models.
References
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Can Large Language Models Reason? A Characterization via 3-SAT
- FOLIO: Natural Language Reasoning with First-Order Logic
- Results on Translating Defaults to Circumscription
- The Power of Beliefs or Translating Default Logic into Standard Autoepistemic Logic
- A Logic Program Characterization of Causal Theories
- On the Relationship between P-log and LPMLN
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples