This paper investigates a puzzling phenomenon in Natural Language Processing: why do Large Language Models (LLMs) still perform well on benchmarks even when the input is "corrupted" (e.g., by removing all nouns or shuffling word order)? The authors set out to determine if this robustness is a sign of memorisation (the model saw the test data during pre-training) or shortcut exploitation (the model is using superficial cues like "not" to guess the answer).
TL;DR
LLMs maintain high performance on NLI benchmarks after severe data corruption not because they have memorised the test sets, but because they exploit superficial dataset "shortcuts." This is confirmed by testing on entirely new, human-validated data where models still fall for the same stylistic traps.
What Problem Does This Paper Address?
In Natural Language Inference (NLI), a model is asked if a "premise" sentence entails a "hypothesis" sentence. Previous research showed that older, fine-tuned models could often guess the correct label even if the input was mangled.
With the rise of LLMs trained on massive web crawls, a new concern emerged: data contamination. If an LLM performs well on a "noun-less" version of a benchmark, is it just because it recognizes the specific sentence from its training data? This paper seeks to disentangle whether LLMs are actually "understanding" something deep, simply remembering the test set, or just getting lucky by spotting annotation artifacts (shortcuts).
Methods
The authors subjected eight LLMs (including Qwen, OLMo, GPT-4o, and DeepSeek) and several fine-tuned encoders (like RoBERTa) to various input perturbations:
- Word-class ablation: Removing all nouns, verbs, adjectives, etc.
- Word-order perturbation: Shuffling the order of words.
- Hypothesis-only: Removing the premise entirely to see if the model can guess the answer from the hypothesis alone.
To account for the fact that LLMs often have an "answer bias" (e.g., preferring to say "entailment" when confused), the authors used centred accuracy, which adjusts scores based on the model's baseline label preferences. They also performed direct contamination checks using exact $n$-gram counts on models with public training data (OLMo).
Datasets
- Standard Benchmarks: GLUE (including MNLI) and three rounds of the adversarial ANLI dataset.
- Fresh Items: The authors created 368 new NLI items based on Wikipedia articles published after September 2026 (past the models' training cut-offs). These were split into "MNLI-style" (containing known shortcuts) and "cue-free" (written to avoid them).
- BioASQ: A biomedical benchmark used as a control because it lacks the typical stylistic shortcuts found in general NLI datasets.
Key Insights and Ideas
- The MNLI vs. ANLI Contrast: Models are significantly more robust to corruption on MNLI than on ANLI. As seen in Figure 1, performance retention on MNLI (36–72%) far outstrips ANLI (14–29%). This suggests MNLI is "leaky" with shortcuts that survive corruption, whereas ANLI was successfully designed to be more rigorous.
- Contamination is not the primary driver: The authors found no correlation between how much of a dataset was present in a model's pre-training data and how robust the model was to perturbations.
- Shortcuts live in function words: When nouns are removed, models still have access to function words like "not," "no," and "never." These are strong indicators of "contradiction" in MNLI. When the authors removed these (the
keep-NOUN+VERBcondition), the models' performance advantage vanished. - Reasoning models are different: Grok 4.7, which "thinks" before answering, was the only model that didn't follow the shortcut pattern, performing better on the harder ANLI dataset than on MNLI.
Key Results
The study demonstrates that "easy" items—those a simple classifier can solve using only the hypothesis—are the most robust to data corruption.
As shown in Figure 2, there is a clear gap in degradation between "easy" and "hard" items. On MNLI, easy items lose significantly less performance when nouns are removed or words are shuffled compared to hard items.
On the Fresh Items, the results were definitive: even on data the models could not have seen, they were 50–75% robust on "MNLI-style" items but much weaker on "cue-free" items. This proves the robustness comes from the style of the benchmark, not memorisation of the content.
Limitations and Future Work
- Verbatim vs. Semantic Contamination: The $n$-gram checks only catch exact matches. If a benchmark exists in the training data in a slightly paraphrased form, these checks might miss it.
- Classifier Strength: The "shortcut" detection relied on a simple logistic regression. A more powerful model might identify even more subtle shortcuts that LLMs are exploiting.
- Reasoning Models: The unique behavior of Grok 4.7 suggests that "Chain of Thought" or reasoning-heavy architectures might finally be moving past simple shortcut exploitation, a promising area for future study.
Summary updated