Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

Joykirat Singh, Akshay Nambi, Vibhav VineetSubmission to ICLR 2025 · Rejected

Introduction

While Large Language Models (LLMs) frequently achieve high accuracy on standard mathematical benchmarks, these scores may mask a fundamental weakness. Recent research suggests that models often rely on memorization rather than robust logical verification. This paper introduces the MWP-MISTAKE dataset to evaluate whether models can truly identify and correct reasoning flaws, moving beyond simple "final answer" metrics.

The MWP-MISTAKE Framework

To move the goalposts from generation to verification, the researchers developed a framework centered on two primary tasks:

  1. Task-1 (T1): Detecting an incorrect reasoning chain, rectifying the specific error, and providing the correct final answer.
  2. Task-2 (T2): Identifying whether a reasoning chain is correct or incorrect and providing the final answer without explicit correction.
Example of the mistake detection and rectification task where GPT-4o successfully identifies a calculation error that GPT-3.5 Turbo misses.

The study utilized a diverse set of error injection methods, including rule-based corruptions (e.g., operator swapping, step reordering) and "natural" errors generated by smaller language models (SLMs) like Llama-2-7b. These were applied across benchmarks ranging from grade-school math (GSM-8K) to advanced competition and college-level problems (MATH, JEEBENCH).

Key Insights: Memorization vs. Reasoning

The evaluation revealed several critical patterns in how current state-of-the-art models handle logical auditing:

  • The "o1" Paradox: Surprisingly, OpenAI’s o1 model, despite its superior problem-solving capabilities, was found to be weaker at mistake detection than GPT-4o. This suggests that increased "thinking" time during generation does not automatically translate to better critical auditing skills.
  • The Memorization Trap: A significant finding was that models often produce the correct final answer even when the provided reasoning steps are nonsensical. This indicates "data contamination," where the model recognizes a problem from its training set and ignores the faulty logic presented in the prompt.
  • The Complexity Gap: Models struggle significantly as problem complexity increases. While they are adept at spotting simple arithmetic errors, their performance plummets on multi-step algebraic or calculus problems found in the JEEBENCH dataset.

Performance Analysis

GPT-4o emerged as the top performer in overall mistake detection, yet the results highlight a stark decline in reliability for advanced mathematics.

F1 scores for GPT-4o and GPT-4 across various mathematical benchmarks, showing a noticeable performance drop on more complex datasets like JEEBENCH.

As shown in the performance data, GPT-4o achieved an overall F1 score of 87% for mistake detection. However, on the advanced JEEBENCH dataset, its accuracy in deriving the correct answer while handling mistakes dropped to just 38%. This underscores the "Achilles' heel" of current LLMs: they can generate solutions but struggle to act as reliable verifiers for complex logic.

Conclusion and Future Directions

The study concludes that mathematical reasoning—specifically the ability to detect and rectify errors—remains a major hurdle for AI. While fine-tuning smaller models (like Llama-3-8b) shows promise for domain-specific error detection, the broader challenge of contamination-proof reasoning persists. Future work must focus on Reinforcement Learning from Human Feedback (RLHF) targeted specifically at verification to bridge this gap and create truly intelligent mathematical assistants.

Summary updated

Abstract

Large Language Models (LLMs) have significantly impacted the field of Math Word Problems (MWPs), transforming how these problems are approached and solved, particularly in educational contexts. However, existing evaluations often focus on final accuracy, neglecting the critical aspect of reasoning capabilities. This work addresses that gap by evaluating LLMs' abilities to detect and correct reasoning mistakes. We present a novel dataset, MWP-MISTAKE, containing MWPs with both correct and incorrect reasoning steps generated through rule-based methods and smaller language models. Our comprehensive benchmarking of state-of-the-art models such as GPT-4o and GPT4 uncovers important insights into their strengths and limitations. While GPT-4o excels in mistake detection and rectification, gaps remain, particularly in handling complex datasets and novel problems. Additionally, we identify concerns with data contamination and memorization, which affect LLM reliability in real-world applications. While OpenAI's O1 model demonstrates 90\% accuracy in reasoning and final answers on complex tasks, it still remains weak in mistake detection. Our findings highlight the need for improved reasoning evaluations and suggest ways to enhance LLM generalization and robustness in math problem-solving.
BibTeX
@misc{singh2024exposing,
  author = {Joykirat Singh and Akshay Nambi and Vibhav Vineet},
  title = {{Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning}},
  year = {2024},
  eprint = {2406.10834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10834},
}
Figure