MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models

Wentian Wang, Sarthak Jain, Paul B. Kantor, Jacob J. Feldman, Lazaros K. Gallos, Hao Wang

Introduction

The rapid advancement of Large Language Models (LLMs) has sparked a fundamental debate: are these systems truly reasoning, or are they merely "stochastic parrots" performing sophisticated pattern matching? While benchmarks like Massive Multitask Language Understanding (MMLU) are widely used to measure intelligence, their reliability is increasingly questioned due to potential data leakage. If a model has seen test questions during training, its high score may reflect memorization rather than comprehension.

To address this, researchers have introduced MMLU-SR (Symbol Replacement). This benchmark tests whether LLMs can maintain reasoning capabilities when familiar technical terms are replaced with arbitrary symbols, provided with their definitions. True understanding should be symbol-invariant; if a model understands a concept, it should be able to apply it regardless of the specific linguistic label used.

The Symbol Replacement (SR) Methodology

The MMLU-SR benchmark is constructed through an automated pipeline with human-in-the-loop verification. The process involves four primary stages:

  1. Term Extraction: Identifying key technical terms across 57 academic subjects using GPT-3.5-turbo.
  2. Definition Generation: Retrieving precise, context-specific definitions for these terms.
  3. Symbol Replacement: Substituting original terms with random dummy words (e.g., "Bard," "Zelly," "Dummy") and appending their definitions to the prompt.
  4. Expert Review: A manual review process to ensure the accuracy and clarity of definitions, correcting approximately 4.2% of the generated content.
Illustration of the Symbol Replacement (SR) process transforming a standard MMLU question into an MMLU-SR variant.

The researchers created three variants of the benchmark to isolate different reasoning challenges:

  • Question Only: Terms are replaced only in the question stem.
  • Answer Only: Terms are replaced only in the multiple-choice options.
  • Question and Answer (Q&A): Terms are replaced in both, requiring the model to map definitions across the entire context.

Key Findings and Performance Degradation

Evaluation across seven major model families—including GPT-4o, Gemini 1.5 Pro, and Llama 3-70B—revealed a significant performance gap between standard MMLU and MMLU-SR. Every tested model experienced a substantial reduction in accuracy.

For instance, GPT-4o's accuracy dropped from 84.5% on the original MMLU to 69.5% on the MMLU-SR Q&A subset. Similarly, Llama 3-70B saw a steep decline from 76.5% to 57.6%. The relative performance drops ranged from 16.59% to 24.71%, highlighting a widespread reliance on memorized terminology.

The Contextual Aid Paradox

Models generally performed better on the "Question Only" subset compared to the "Answer Only" subset. The researchers hypothesize that when answer choices remain in their original form, they act as "contextual aids," allowing models to reverse-engineer the meaning of dummy words in the question. Conversely, the brevity of answer choices provides fewer anchors for the model to map dummy words back to its pre-trained knowledge.

Reasoning vs. Recognition

A critical observation was the failure of Chain-of-Thought (CoT) prompting to bridge the gap. Even when models correctly identified the underlying concept, they frequently struggled to maintain the symbolic mapping throughout the reasoning chain, ultimately selecting the wrong dummy word.

Example of a model failing to reason correctly when both question and answer terms are replaced with dummy symbols.

Discussion and Future Work

The results from MMLU-SR suggest that current LLMs are heavily tethered to specific linguistic patterns. While they show more resilience in STEM subjects—likely due to the structural nature of mathematical reasoning—their performance in the Humanities suffers greatly when familiar labels are removed.

Future research will explore more complex variations, such as shuffling definitions or introducing entirely synthetic concepts. By stripping away the "crutch" of familiar terminology, MMLU-SR provides a more rigorous measure of true language understanding and sets a new standard for evaluating robust reasoning in artificial intelligence.

Summary updated

BibTeX
@misc{wang2024mmlu,
  author = {Wentian Wang and Sarthak Jain and Paul B. Kantor and Jacob J. Feldman and Lazaros K. Gallos and Hao Wang},
  title = {{MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models}},
  year = {2024},
  eprint = {2406.15468},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.15468},
}
Figure