Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements

Guangxiang Zhao, Saier Hu, Xiaoqi Jian, Wu Jinzhu, Yuhan Wu, Lin Sun, Xiangzheng ZhangEMNLP 2025

TL;DR

This paper introduces a "Generalization Stress Test" to reveal that even state-of-the-art Large Language Models (LLMs) struggle to generalize when faced with minor, content-preserving changes to questions, such as altering option length or replacing irrelevant nouns. Despite high benchmark scores, LLMs exhibit significant accuracy drops, suggesting they rely on superficial cues rather than robust reasoning.

What Problem Does This Paper Address?

Large Language Models (LLMs) have demonstrated impressive performance on various benchmarks, leading to claims of near-human generalization capabilities. However, a critical question remains: do these high scores reflect true understanding and generalization, or are LLMs simply exploiting superficial patterns and biases present in the training data and evaluation benchmarks? Existing benchmarks might be susceptible to data contamination or may not adequately probe the models' ability to handle slight variations in input. This paper aims to address this by rigorously testing LLMs' generalization under controlled, minor perturbations that should ideally not affect performance if true understanding is present.

Methods

The authors propose a "Generalization Stress Test" framework that applies three types of minor, content-preserving perturbations to existing benchmark tasks:

  • Altering Option Length: This involves either lengthening the correct option (RL) or lengthening incorrect options (WL) without changing the core meaning of the question or answer. This tests for biases related to option length.
  • Changing Problem Types: Multiple-choice questions (MCQ) are converted into boolean questions (BQ). This tests if LLMs can adapt their reasoning to different output formats while retaining the underlying knowledge.
  • Replacing Irrelevant Nouns: Nouns that are semantically irrelevant to the problem's solution are replaced with other nouns. This assesses whether LLMs can ignore superficial changes and focus on the essential information.

The authors also conducted ablation studies to investigate potential causes for the observed performance drops, including analyzing attention patterns and testing simple mitigation strategies like fine-tuning and Chain-of-Thought (CoT) prompting.

Datasets

The experiments were conducted on established benchmarks:

  • MMLU (Massive Multitask Language Understanding): A comprehensive benchmark covering 57 diverse tasks.
  • ARC-Challenge (AI2 Reasoning Challenge): A challenging benchmark focusing on common sense and reasoning.
  • GSM8K (Grade School Math 8K): A dataset of multi-step math word problems.

The paper does not introduce new datasets but rather modifies existing ones through the proposed perturbation methods.

Key Insights and Ideas

  • Superficial Cues Over Robust Reasoning: The most striking finding is that LLMs, including top-tier models like GPT-4o, exhibit significant performance degradation when faced with minor, semantically irrelevant changes. This strongly suggests they rely on superficial cues (like option length) rather than forming abstract, generalizable representations.
  • Length Bias is Pervasive: LLMs show a clear bias towards longer options, particularly when the correct option is lengthened. Conversely, lengthening incorrect options can sometimes lead to improved performance, hinting at a strategy of picking the most "prominent" or detailed option.
  • Format Agnosticism is Lacking: Converting multiple-choice questions to boolean questions leads to substantial accuracy drops, indicating that LLMs struggle to generalize their knowledge across different output formats.
  • Irrelevant Information is a Distractor: Replacing semantically irrelevant nouns, even with words that don't alter the problem's core logic, consistently degrades performance. This highlights a fragility in their ability to filter out noise.
  • Simple Mitigations Fall Short: While fine-tuning and CoT prompting can offer some improvement, they do not fully resolve the observed vulnerabilities, suggesting that more fundamental architectural or training changes might be needed.

Key Results

  • Qwen2.5 1.5B: MMLU score dropped from 89 to 36 when option lengths were changed.
  • GPT-4o: Experienced a 25-point accuracy loss when problem types were changed and a 6-point drop across all three modification categories on MMLU.
  • Across various models and benchmarks, accuracy drops of 10-30 points were observed for different perturbation types. For instance, in Table 1, RL (lengthening right option) often increases scores for many models, while WL (lengthening wrong option) generally decreases them.
  • In Table 3, changing from MCQ to BQ on MMLU resulted in significant drops, with Qwen2.5 1.5B and Llama3.1 8B scoring only half the points in the "both true" setting.
  • Table 4 shows consistent performance degradation across models when irrelevant nouns were replaced on GSM8K.

Limitations and Future Work

  • Focus on Non-CoT LLMs: The study primarily focuses on non-chain-of-thought LLMs, and the authors acknowledge that emerging proprietary LLMs (like Gemini and Claude) are also sensitive to these changes, but a full analysis of CoT models is outside the scope.
  • Need for Stronger Defenses: The paper suggests that simple mitigations like fine-tuning or CoT are insufficient. Future work should explore stronger defenses, such as adversarial training or architectural modifications, to build more robust LLMs.
  • Understanding Attention: While the paper shows that attention mechanisms are affected by option length, further research is needed to fully understand how these mechanisms contribute to the observed biases.
  • Broader Perturbations: The authors' "Generalization Stress Tests" provide a valuable framework, and future work could expand this by exploring a wider range of perturbations and more complex scenarios.

Summary updated

BibTeX
@inproceedings{zhao2025large,
  author = {Guangxiang Zhao and Saier Hu and Xiaoqi Jian and Wu Jinzhu and Yuhan Wu and Lin Sun and Xiangzheng Zhang},
  title = {{Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure