Does the Answer Move? Graded Negation Edits as a Test for Benchmark Memorization

When a language model scores well on a public benchmark, we cannot tell whether it solved the items or saw them during training. This leakage of test data into training is called benchmark contamination. Most black-box detectors ask whether the model's answer stays the same when an item is reworded without changing its meaning. That question has a known weakness. A model trained on paraphrases of the test set also learns to give the same answer across paraphrases, so it passes these checks. We propose asking the opposite question. Make a small edit that changes the item's meaning by a known, graded amount, and check whether the answer moves by that amount.

Graded negation is the natural tool for such edits. Negation in real language works like a dial more than a switch. "Not excellent" means mediocre, not terrible. "Not bad" means acceptable, not brilliant. "Hardly" and "barely" dampen sentiment rather than reverse it. A model that has memorized "this review → positive" should keep returning the memorized label when the review is changed to "hardly a delight," even though the right answer has shifted toward neutral. A model that reads the text should follow the shift. Graded edits also avoid a shortcut that plain "insert not" edits leave open. Current models flip their answer whenever they see a negation word, so a binary negation flip can be passed by spotting the cue, whether or not the model has memorized the item.

Core Claim

Our hypothesis is that contaminated items show label stickiness under meaning-changing graded negation edits. Stickiness means the model fails to move its prediction in proportion to the known semantic shift. We measure it against the same model's behavior on matched items that cannot be in any training set. This signal should survive paraphrase-based evasion, because paraphrase training teaches the model to ignore surface changes, and that habit makes it more blind to small meaning changes, not less. The specific novelty is graded, meaning-altering edits calibrated against fresh controls. Meaning-preserving perturbation and binary label flips are the tools existing detectors already use, and they are the comparison points here.

Where This Sits

The negation literature supplies the mechanism. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank built the Stanford Sentiment Treebank (SST), which labels sentiment at the phrase level. It showed that negating a negative phrase pulls sentiment toward neutral rather than to the positive extreme. An Empirical Study on the Effect of Negation Words on Sentiment showed that negators have word-specific shifting strengths. It also showed that a reversal heuristic fits SST negated phrases poorly, with a mean absolute error of 0.1480 against 0.1062 for its compositional model. On Positivity Bias in Negative Reviews shows that real negative reviews often use negated positives, so these edits produce natural text rather than adversarial noise. We reuse these findings as a recipe for edits whose target shift is known in advance. We do not build a new composition model.

Thunder-NUBench plays two roles. First, it motivates the design. When zero-shot models err on it, they pick the "local negation" distractor over 70% of the time, which is shallow cue detection. That is exactly why binary flips are a weak contamination probe. Second, it is an audit target. Its supervised fine-tuning on 3,772 negation pairs raises accuracy sharply, and nobody has checked whether that gain is learned scope or memorized templates.

The contamination literature supplies the problem and the competitors. CAP compares consistency under shuffling, distractor edits and paraphrase. It distinguishes contamination from honest fine-tuning by comparing consistency on train and test splits, and we borrow that comparison. PEARL measures output change under random bit-flip noise, and it loses power outside verbatim-completion tasks. The Data Contamination Quiz asks the model to pick the original text out from synonym swaps, and it explicitly cannot catch paraphrased contamination. All three use edits that keep meaning or destroy it at random; none uses edits whose meaning change is graded and known.

The constraint that shapes our design comes from two papers. Evading Data Contamination Detection for Language Models is (too) Easy shows that training on LLM paraphrases of test items keeps up to 15% of the score boost. It also pushes sample-level detection to about 1% true-positive rate at 1% false-positive rate. Does Data Contamination Detection Work (Well) for LLMs? shows that perplexity detectors sit near 0.50 AUC (area under the ROC curve; 0.50 is chance) for pretraining contamination on Pythia and OLMo-2, while reaching 99.4 in a supervised fine-tuning setting.

DyVal and INVITE generate fresh test items for formal logic and for binary false-premise questions. We borrow their idea of fresh generation, but only to build the matched clean controls. Our goal is auditing existing benchmark scores, not replacing benchmarks.

Research Program

Edit construction. For each benchmark item whose label depends on polarity or valence, we produce three kinds of edits. Shift edits insert or swap a graded modifier ("excellent" → "not excellent"; "dull" → "not dull"; "a delight" → "hardly a delight"). Their target label is derived from SST phrase-level statistics for that negator–argument pattern, and a human-rated subset checks calibration. Invariance edits add a local negation inside an embedded clause that should leave the label unchanged, following Thunder-NUBench's local-negation definition. Binary flips insert a main-clause "not," and they serve as the ablation.

Stickiness statistic. For model $m$ and item $i$, let $s_i$ be the fraction of shift edits on which $m$ keeps the original predicted label even though the target label changed. We score an item by

$$ z_i = \frac{s_i - \mu_m(c_i)}{\sigma_m(c_i)}, $$

where $\mu_m(c_i)$ and $\sigma_m(c_i)$ are the mean and standard deviation of stickiness for model $m$ on freshly written items. These fresh items match item $i$'s construction pattern $c_i$, meaning the same negator, argument polarity and syntactic depth. They are written after every model's cutoff and never published.

The normalization is the core of the method. It separates "this model handles 'hardly' poorly in general" from "this model handles 'hardly' poorly on this particular item." Benchmark-level contamination is estimated from the excess of high $z_i$ over the fresh-item distribution. Invariance edits serve as a guard. A model that is simply erratic will move on invariance edits too, whereas a memorizing model should be stable on them and sticky on shift edits.

Ground truth. Detection claims need known contamination. We fine-tune and continue-pretrain open models (Pythia and OLMo-2, whose training data is public) under five conditions. These are no exposure, verbatim test exposure, paraphrase-only exposure in the style of the evasion paper, honest exposure to the training split only, and exposure mixed into the pretraining corpus at low frequency. Only after validating on these models do we audit closed models on SST, Thunder-NUBench, AG News and the WNLI natural-language-inference set.

Baselines

The naive reference point is perplexity thresholding on the original items. It is run on identical items and models, with the threshold tuned on the same held-out split as ours. The cited evaluation reports AUC near 0.50 for pretraining contamination (Does Data Contamination Detection Work (Well) for LLMs?), and our pretraining-mixture condition should reproduce that.

The strongest existing method is CAP. It is the closest competitor because it already combines black-box perturbations with a train/test direction check that guards against mislabeling honest fine-tuning. We run all three of its perturbation families with the same query budget per item as our method. PEARL and the Data Contamination Quiz are run the same way. The file reports no AUC for CAP, PEARL or the Quiz on these benchmarks, so no reported result was found on Lacuna for those numbers, and we will measure them ourselves.

The figure to beat comes from Evading Data Contamination Detection: about 1% true-positive rate at 1% false-positive rate for existing sample-level detectors on paraphrase-contaminated models. We would count a true-positive rate at 1% false-positive rate several times higher on the paraphrase-only condition as beating it, provided we stay competitive with CAP on verbatim contamination. A gain only on verbatim contamination would not count.

Two ablations isolate the key ingredient. The binary-flip ablation replaces graded edits with main-clause "not" insertion. The uncalibrated ablation uses raw $s_i$ without the fresh-item normalization. If either ablation matches the full method, the corresponding claim fails.

Why not just…?

Why not just use binary negation flips, like contrast sets? Because of the cue shortcut. Thunder-NUBench shows that models treat any "not" as a reversal, so a memorizing model can pass a binary flip by reacting to the word. Graded edits ("not bad" → acceptable) and local negations punish that reflex. This is a real empirical question, though, and the binary-flip ablation settles it. If binary flips detect paraphrase contamination equally well, the graded component adds nothing, and we will report that.

Why not just use CAP's paraphrase consistency? Paraphrase-trained contamination is built to be consistent under paraphrase. The evasion paper shows paraphrase-style augmentation defeats existing sample-level detectors. Our paraphrase-only condition tests directly whether CAP's signal disappears there while stickiness remains.

Why not just build a fresh benchmark and stop auditing old ones? For new evaluations, fresh generation in the style of DyVal is the right answer, and we use it for our controls. But papers keep reporting SST, AG News and Thunder-NUBench numbers, and only an audit can say what those numbers mean. Fresh generation also cannot answer whether Thunder-NUBench's fine-tuning gains are memorized templates.

Why not just use perplexity? In the pretraining regime, the cited evaluation finds it at chance. We keep it as the floor baseline.

Decisive Checks

Four results would decide the proposal. On the controlled open models, stickiness $z$ must separate the paraphrase-only condition from no exposure and from honest training-split exposure. The train-split control is essential: if honest fine-tuning also raises stickiness, the signal reflects domain fitting rather than memorization, and the method fails. The fresh-item calibration must keep false positives near nominal on uncontaminated models across model sizes. Small models handle negation worse, according to Thunder-NUBench's scaling results, so this check matters most there. The binary-flip ablation must perform clearly worse than graded edits on paraphrase contamination.

Finally, we test an adaptive attacker who also trains on negated variants. To keep the score boost, that attacker needs correct labels for the edited items, which means solving the task itself. We expect the inflated score to shrink as evasion of our probe improves, and we will measure that trade-off directly rather than assume it.