MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models

Wentian Wang, Paul B. Kantor, Jacob J. Feldman, Lazaros K. Gallos, Hao Wang, Wang, Hao

Abstract

We propose MMLU-SR, a novel dataset designed to measure the true comprehension abilities of Large Language Models (LLMs) by challenging their performance in question-answering tasks with modified terms. We reasoned that an agent that "truly" understands a concept can still evaluate it when key terms are replaced by suitably defined alternate terms, and sought to differentiate such comprehension from mere text replacement. In our study, we modified standardized test questions by replacing a key term with a dummy word along with its definition. The key term could be in the context of questions, answers, or both questions and answers. Notwithstanding the high scores achieved by recent popular LLMs on the MMLU leaderboard, we found a substantial reduction in model performance after such replacement, suggesting poor comprehension. This new benchmark provides a rigorous benchmark for testing true model comprehension, and poses a challenge to the broader scientific community.

TL;DR

MMLU-SR is a new benchmark that "stress-tests" whether Large Language Models (LLMs) actually understand concepts or are just memorizing text. By replacing key terms with dummy words and their definitions (e.g., replacing "enzyme assay" with "Hat"), researchers found that even top-tier models like GPT-4o and Gemini 1.5 Pro suffer significant performance drops, revealing a heavy reliance on memorized patterns over abstract reasoning.

What Problem Does This Paper Address?

As LLMs dominate standardized tests like the MMLU (Massive Multitask Language Understanding), a skeptical question persists: Are these models truly reasoning, or are they "stochastic parrots" that have simply memorized the test data during training?

A hallmark of human intelligence is symbolic flexibility—the ability to understand a concept regardless of the specific word used to describe it. If an LLM truly understands a concept, it should be able to solve a problem even if a key term is replaced by a nonsense word, provided a clear definition is given. The authors address the "brittleness" of LLM knowledge and the potential for data leakage by introducing a "Symbol Replacement" (SR) challenge.

Methods

The researchers developed an automated pipeline to transform the original MMLU dataset into MMLU-SR through four main steps: term extraction, dictionary creation, symbol replacement, and scenario construction.

Comparison of standard MMLU and the MMLU-SR symbol replacement process.

They created three distinct subsets to isolate where the reasoning breaks down:

  • Question Only (QO): Only terms in the question are replaced.
  • Answer Only (AO): Only terms in the multiple-choice answers are replaced.
  • Question and Answer (QA): Terms are replaced in both, requiring the model to maintain a "mental map" of the new definitions throughout the entire prompt.

Key Insights and Ideas

  • The "Contextual Crutch": Models performed best on the "Question Only" subset. This suggests that if the answer choices remain in their original form, the model can use them as a hint to "reverse-engineer" what the dummy word in the question actually means.
  • Labels vs. Concepts: Performance plummeted most in the "Answer Only" and "QA" subsets. This indicates that LLMs rely heavily on the label of a concept (the word itself) to trigger their internal knowledge, rather than being able to reason from a description of that concept.
  • Reasoning vs. Pattern Matching: Even when using Chain-of-Thought, models often correctly identified the definition but then mapped it to the wrong dummy word or hallucinated a connection to a different concept. This highlights a failure in "binding" definitions to new symbols during active reasoning.
Example of an LLM failing a Question and Answer (QA) replacement task despite correctly identifying definitions.
  • STEM Resilience: STEM subjects generally saw smaller performance drops than the Humanities. This is likely because mathematical and logical structures are inherently more symbolic and less dependent on specific linguistic "anchors."

Key Results

The study evaluated seven major model families, including GPT-4, Gemini, and Llama 3. The results were consistent: Symbol replacement hurts.

  • GPT-4o: Dropped from an average of 84.5% on the original MMLU to 69.5% on the MMLU-SR (QA) subset.
  • Llama3-70B: Saw a significant decline from 76.5% to 57.6%.
  • Relative Accuracy Drop: Across all models, the "Question and Answer" modification caused a relative performance drop of 16% to 32%.

Limitations and Future Work

While the process was largely automated, some manual human correction was required for definitions. Future work could extend this "stress test" to coding tasks or multimodal models to see if the reasoning gap persists across different domains. By providing MMLU-SR, the authors have given the AI community a tool to move beyond "leaderboard chasing" and toward a more rigorous evaluation of whether LLMs are truly thinking.

Summary updated

Abstract

We propose MMLU-SR, a novel dataset designed to measure the true comprehension abilities of Large Language Models (LLMs) by challenging their performance in question-answering tasks with modified terms. We reasoned that an agent that "truly" understands a concept can still evaluate it when key terms are replaced by suitably defined alternate terms, and sought to differentiate such comprehension from mere text replacement. In our study, we modified standardized test questions by replacing a key term with a dummy word along with its definition. The key term could be in the context of questions, answers, or both questions and answers. Notwithstanding the high scores achieved by recent popular LLMs on the MMLU leaderboard, we found a substantial reduction in model performance after such replacement, suggesting poor comprehension. This new benchmark provides a rigorous benchmark for testing true model comprehension, and poses a challenge to the broader scientific community.
BibTeX
@misc{wang2024mmlu,
  author = {Wentian Wang and Paul B. Kantor and Jacob J. Feldman and Lazaros K. Gallos and Hao Wang and Wang, Hao},
  title = {{MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models}},
  year = {2024},
  eprint = {2406.15468},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.15468},
}
Figure