Sentence Splitting as a One-Sided Test for Benchmark Contamination

When a language model has seen a benchmark's test questions during training, its score partly reflects memory rather than skill. The usual fix is to reword the questions and see whether the score falls. This has a known flaw: a free rewrite can make a question harder, change its facts, or change its domain, so a score drop no longer cleanly means "the model had memorized it." We propose a rewrite that only makes questions easier. We split each question into short, standalone sentences. Every number, entity and relation stays the same; only the nesting of clauses is undone. A model that actually reasons should do at least as well on the split version. A model that recalls the original wording loses its cue. Because the honest model's expected change has a known sign, any drop becomes evidence of memorization rather than of difficulty.

This matters most where current tools fail. Detectors based on token likelihood collapse to near chance once a contaminated model receives reinforcement-learning or chain-of-thought post-training (On The Fragility of Benchmark Contamination Detection in Reasoning Models). That leaves behavioral tests, and those rely on unconstrained rewrites by a large language model. Syntactic simplification is the task of restructuring a complex sentence into simple clauses while keeping its meaning, often called split-and-rephrase. It supplies a controlled rewrite whose effect on difficulty is predictable. That predictability is the piece the contamination literature is missing.

Core Claim

The claim is that a purely structural, fact-preserving decomposition of benchmark items yields a contamination estimate whose confound runs in one direction. Prior rewrite-based methods mix memorization loss with unknown difficulty shifts. Decomposition removes the shift in difficulty, or pushes it in the honest model's favor. We test this claim on models deliberately contaminated under the post-training regimes where likelihood detectors fail. We also test the claim's weak point directly: whether splitting a question can by itself hurt clean models, through broken pronoun references or lost cohesion.

We do not claim that a hand-built parser beats a language model at splitting sentences. The ingredient we isolate is the constraint: no lexical change, no added or deleted facts, and lower clausal depth. Combined with a control-set difference, that constraint is what makes the estimate interpretable. Who produces the split is a secondary question, and one ablation settles it.

Relation to Prior Work

Three papers motivate this work by rewriting benchmarks to expose contamination, and all three use unconstrained generation. Inference-Time Decontamination uses a likelihood detector to flag memorized items, has GPT-4 rewrite their background stories, and re-checks the result. CLEAN-EVAL combines paraphrasing with multi-language back-translation and then filters candidates by semantic similarity. ConStat compares a model's score on the original benchmark against GPT-4-Turbo-rephrased reference sets. It calibrates the comparison with a fleet of reference models, so it can separate inflation from ordinary difficulty differences.

We borrow ConStat's control logic but replace its rewriter, and we drop ITD's reliance on likelihood detectors. The fragility paper shows those detectors are unusable for reasoning models: average precision-style detection (AUROC) falls from 89.1% to 50–58%. That paper constrains the design: our probe must be behavioral.

Back to Square One strips clauses from Winograd items to expose shortcut learning. Its edits are destructive and remove information. Ours preserve all information, and we use clause deletion only as an ablation.

The method itself comes from the split-and-rephrase line of work, and this work supplies both the technique and the caution. The technique comes from three sources. Graphene recursively pulls subordinate clauses out into core and context propositions using hand-written patterns. Leveraging Linguistic Structure For Open Domain Information Extraction splits clauses along dependency parses and resolves implicit subjects. TransBooster replaces complex parts of a sentence with placeholders, which handles references that cross the new sentence boundaries.

The caution comes from Small but Mighty. It shows that neural splitters produce perfect rewrites only 16.7% of the time on legal text, against 63.3% for humans. It also shows that BLEU, a word-overlap score, fails to track whether facts are preserved. Our pipeline therefore verifies every split rather than trusting the generator. The same paper documents that models are fragile to dense syntax. That fragility is the confound we exploit: removing density should help clean models, not hurt them.

Research Program

We first build a decomposer for benchmark question text. It uses dependency and semantic-role parses to find split points: coordination, relative clauses, appositives, participial and subordinate clauses. It extracts the embedded clauses as standalone sentences. It replaces pronouns that would dangle with their antecedent noun phrases. It keeps the final question sentence last.

Each output passes an automatic check before it is used. The multiset of numbers and content words must match the original, apart from the inserted antecedents. The pipeline then extracts open information extraction triples, i.e. subject–relation–object facts, from both versions, and the propositions must match. Items that fail are dropped and logged. We report the drop rate per benchmark so that coverage is visible.

As a second producer, an LLM is prompted to "split into simple sentences without changing any word choice." Its outputs pass through the same verifier. A human audit on a sample uses the six criteria from Small but Mighty: sensicality, grammaticality, fact preservation, fact integrity, split placement and split sufficiency. We also run both producers on Wiki-BM and Cont-BM, so that split quality is measured against human references and not only on benchmark text.

The estimator is a difference-in-differences: a comparison of the score change on suspect items against the score change on items known to be clean. For a model $m$ and an item set $X$, let $\Delta_m(X)$ be accuracy on the original items minus accuracy on their decomposed versions. With $S$ the suspect items and $C$ the control items, the contamination estimate is

$$\hat\delta_m = \Delta_m(S) - \Delta_m(C).$$

The control items $C$ are known to be clean. They are a held-out half in controlled experiments; for real audits they are freshly written items of matching style. For an uncontaminated model, both $\Delta$ terms should be near zero or negative, and $\hat\delta_m \approx 0$. Memorization makes $\Delta_m(S)$ positive.

Following ConStat, we also fit the expected $\Delta$ as a function of original-item accuracy across a fleet of models believed to be clean. This yields a significance test, not just a point estimate.

Baselines

The naive reference point is to take the benchmark score at face value, which is equivalent to an estimate of $\hat\delta_m = 0$. Next to it we report a likelihood detector as the reference the field still uses. Following the fragility paper's protocol, this is LiRA, a membership test that compares the audited model's likelihood against reference models, alongside Min-K% Prob. Its Stage I result is 89.1% AUROC before reinforcement learning, falling to 50–58% for all ten methods it tests in Stage II. We expect the same pattern and include it to show that the regime is a hard one.

The strongest competitor is ConStat with GPT-4-Turbo rephrased reference sets. It already corrects for difficulty with a model fleet, and it reports 89% sensitivity on syntax-contaminated models and 98% on sample-contaminated models. It also gives δ ≈ 8.2% on GSM8K for Mistral-7b-v0.1. We run it on exactly the same contaminated models, the same reference fleet and the same item budget. The only change is the source of the reference set: ConStat's rephrasings versus our verified decompositions.

ITD is the rewrite-based alternative. It reports accuracy drops of 22.9% on GSM8K and 19.0% on MMLU for an intentionally contaminated Llama-2, versus 0.5% and 3.6% for Mistral-7b. We run its GPT-4 rewrite step alone, without its likelihood-based detection and assurance steps, because those are invalid in our regime. CLEAN-EVAL's back-translation-plus-filter pipeline is the third rewrite baseline. It reports a gap reduction on QQP, a duplicate-question benchmark, from 54.33 to 26.33.

Two ablations isolate the key ingredient. The first swaps the decomposer for clause deletion, the destructive probe used in Back to Square One. The second swaps it for LLM splitting without the verifier.

To beat ConStat, our probe must show three things across seeds. First, a higher detection AUROC at equal false-positive rate on the post-reinforcement-learning and chain-of-thought-contaminated models. Second, a smaller absolute error between $\hat\delta$ and the true inflation, which we measure against an uncontaminated twin of each model. Third, a near-zero mean estimate on the clean fleet. A tie on detection paired with lower false-positive estimates on clean models would still count as a win, since that is the property the one-sided argument predicts.

Why not just…?

Why not just have GPT-4 paraphrase the questions, as ITD and ConStat do? Free paraphrases change wording, domain and sometimes difficulty. The Split and Rephrase work found that end-to-end neural models without structural scaffolding lag pipeline systems badly: BLEU 48.92 versus 78.77 on WebSplit, a benchmark of sentences generated from structured facts (Split and Rephrase). That gap suggests uncontrolled rewrites drift. The direct test is the clean-fleet check. If ConStat's rephrasings produce $\hat\delta$ estimates for clean models that are as tight as ours, the one-sided advantage is not real and we will say so.

Why not just back-translate? CLEAN-EVAL needed an extra semantic filter to discard drifted translations. Back-translation also does not reduce clausal depth, so it carries no sign guarantee. It is run as a baseline under the same clean-fleet test.

Why not just ask an LLM to split the sentences? This is a real threat, and it may well match the parser. We treat it as a producer and not as a rival idea. If verified LLM splits perform as well as parser splits, the contribution narrows to the constraint plus the control-difference estimator, which is the part we claim. If unverified LLM splits perform worse, the verifier is what matters.

Why not just retire contaminated benchmarks for fresh ones like LiveBench? Fresh benchmarks also leak eventually, and they cannot score past models on the historical benchmarks those models were compared on. We use freshly written items as the control set $C$ rather than as a replacement.

Proposed Experiments

The core experiment uses controlled contamination on GSM8K, whose 1,319-item test set consists of multi-clause word problems where decomposition has room to act. We take open models in the families the cited papers used: Llama-2/3-scale, Mistral-7b, Phi-3-mini, and DeepSeek-R1-Distill-Llama-8B. We fine-tune each on a random half of the test set, which gives known contaminated and clean halves and an uncontaminated twin.

Each model then goes through the fragility paper's two regimes. In the first, contamination via supervised fine-tuning is followed by clean GRPO reinforcement learning. In the second, the model is contaminated with chain-of-thought traces. We repeat the protocol on ARC-Challenge. We also run it on MMLU, whose questions are often single clauses. There we expect little room to decompose and weak detection, and we report this as the method's coverage boundary rather than hide it. For real-world audits, we report $\hat\delta$ for public models alongside ConStat's published estimates.

No reported result was found on Lacuna for any decomposition-based contamination probe, so every number on our side is new.

Two checks decide the outcome. The first is the sign test. On the clean fleet and on the clean halves, decomposed accuracy must not fall significantly below original accuracy. If it does, through coreference breaks or lost cohesion, the one-sided argument fails. The method then reduces to a ConStat variant, with value only if it improves calibrated accuracy. The second is the fragility test. On the second-regime chain-of-thought-contaminated models, where every likelihood detector sits at 50–58% AUROC, the decomposition probe must recover detection well above chance and track the true inflation. If it does not, then memorized reasoning patterns survive the change in sentence structure. That would be a substantive negative result about what reasoning-model contamination actually stores.