Evaluating Robustness of LLMs to Numerical Variations in Mathematical Reasoning

Yuli Yang, Hiroaki Yamada, Takenobu TokunagaNAACL 2025

TL;DR

Large Language Models (LLMs) show impressive performance on math benchmarks, but this paper reveals they often rely on surface patterns rather than true reasoning. By introducing numerical variations into math word problems, the authors demonstrate that LLMs' performance significantly degrades, especially with larger numbers, highlighting a critical weakness in their arithmetic capabilities.

What Problem Does This Paper Address?

While LLMs have achieved high accuracy on mathematical reasoning benchmarks like GSM8K and MATH, there's a growing concern that this success might stem from memorizing patterns in the training data rather than genuine mathematical understanding. Simply perturbing superficial elements like names or quantities in a problem shouldn't break a model that truly reasons. However, previous studies evaluating this robustness had limitations: they used small variations, didn't explore the impact of digit size, and struggled to distinguish between calculation errors and reasoning errors. This paper aims to address these gaps by proposing a scalable method to generate diverse numerical variations and a framework for automated error analysis to pinpoint the root causes of LLM failures in mathematical reasoning.

Methods

The authors introduce two key innovations:

  1. GSM-ALT Dataset Generation: They developed a template-based approach to create a large-scale dataset of numerical variations from existing math word problems (MWPs).
  2. Question Abstraction: They first abstract numerical values in a question and its solution into variables (e.g., x, y, z). This forms a "question template."
    • Variant Generation: These templates are then populated with new numerical values within specified ranges (Easy: 1-99, Hard: 1-9,999) to create variants. Constraints are manually defined for each template to ensure the generated variants remain valid and sensible (e.g., quantities must be positive).
  3. Scalability: This method allows for generating thousands of variants from a single original question, creating a comprehensive testbed.
  4. Automated Error Classification Framework: To understand why LLMs fail, they developed a framework to distinguish between calculation errors and reasoning errors.
  5. Abstraction of Predicted Solutions: A separate LLM (Qwen2-math-72b-instruct) is used to transform the LLM's predicted solution into an abstracted form, mirroring the template's variables.
  6. Error Identification: By comparing the abstracted predicted answer with the gold abstracted answer from the template, they can infer the error type. If the abstracted answer is correct but the original was wrong, it suggests a calculation error. If the abstracted answer is also incorrect, it points to a reasoning error.

Datasets

  • GSM8K: Used as the base dataset for generating templates. It consists of primary and secondary school math word problems.
  • GSM-ALT: A newly constructed dataset. It contains 2,000 variants (1,000 Easy, 1,000 Hard) for each of the 250 sampled GSM8K templates, resulting in a total of 250,000 variants.

Key Insights and Ideas

  • LLMs are Fragile to Numerical Perturbations: The most striking finding is the significant performance drop across all tested LLMs when faced with numerical variations, even in the "Easy" set with smaller numbers. This suggests their "reasoning" is highly dependent on the specific numbers seen during training.
  • Larger Numbers Exacerbate Weaknesses: The performance degradation is much more pronounced in the "Hard" variant set (larger numbers). This indicates that LLMs struggle more with complex arithmetic operations and potentially with handling larger numerical inputs.
  • Calculation Errors Dominate: The error analysis reveals that as numerical variations increase, the proportion of calculation errors rises significantly, especially in the Hard set. This strongly suggests that LLMs' fundamental arithmetic capabilities are a major bottleneck, rather than their ability to generate logical steps.
  • Reasoning Errors Still Present: While calculation errors become more prevalent, reasoning errors also increase in the variant sets compared to the base dataset, indicating that numerical changes can also disrupt the logical flow of problem-solving.

Key Results

  • Accuracy Drop: For instance, Llama-3.1-8b-Instruct dropped from 0.908 accuracy on the base GSM8K to 0.736 on the Easy GSM-ALT and 0.345 on the Hard GSM-ALT. Wizardmath-7b-v1.1 showed a similar trend, dropping from 0.868 to 0.629 (Easy) and 0.347 (Hard).
  • Error Type Shift: In the base set, most errors were reasoning errors. However, in the Easy and Hard variant sets, calculation errors became the dominant type, accounting for over 69% of errors in the Hard set for some models.
  • Error Classification Success: The automated error classification framework achieved a 90% success rate in correctly transforming predicted solutions into abstracted forms, validating its utility.

Limitations and Future Work

  • Manual Template Creation: The process of creating question templates was manual, as fully automating the "symbolization" of numbers and ensuring variable consistency is challenging.
  • Unrealistic Variants: Some generated variants might contain unrealistic number combinations (e.g., apples costing $1000), though the authors argue this doesn't necessarily invalidate the reasoning evaluation.
  • Limited Reasoning Error Granularity: The error classification framework distinguishes between calculation and reasoning errors but doesn't delve into the specific types of reasoning failures.
  • Future Work: The authors suggest extending the error analysis framework to cluster and aggregate reasoning patterns, which would require significant human resources. They also propose further investigation into how numerical variations influence different reasoning patterns.

Summary updated

BibTeX
@inproceedings{yang2025evaluating,
  author = {Yuli Yang and Hiroaki Yamada and Takenobu Tokunaga},
  title = {{Evaluating Robustness of LLMs to Numerical Variations in Mathematical Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure