Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

Joykirat Singh, Akshay Nambi, Vibhav Vineet

Abstract

Large Language Models (LLMs) have been applied to Math Word Problems (MWPs) with transformative impacts, revolutionizing how these complex problems are approached and solved in various domains including educational settings. However, the evaluation of these models often prioritizes final accuracy, overlooking the crucial aspect of reasoning capabilities. This work addresses this gap by focusing on the ability of LLMs to detect and correct reasoning mistakes. We introduce a novel dataset MWP-MISTAKE, incorporating MWPs with both correct and incorrect reasoning steps generated through rule-based methods and smaller language models. Our comprehensive benchmarking reveals significant insights into the strengths and weaknesses of state-of-the-art models, such as GPT-4o, GPT-4, GPT-3.5Turbo, and others. We highlight GPT-4o's superior performance in mistake detection and rectification and the persistent challenges faced by smaller models. Additionally, we identify issues related to data contamination and memorization, impacting the reliability of LLMs in real-world applications. Our findings emphasize the importance of rigorous evaluation of reasoning processes and propose future directions to enhance the generalization and robustness of LLMs in mathematical problem-solving.

Introduction

The AI community has recently celebrated LLMs for achieving over 90% accuracy on standard math benchmarks like GSM-8K. However, "final answer accuracy" can be a deceptive metric. In critical applications like AI tutoring, a model must do more than provide the right answer; it must understand and verify the logical process. Current evaluations often ignore whether a model can spot a subtle logical error, potentially masking a reliance on data memorization rather than true problem-solving skills.

Methodology: The MWP-MISTAKE Dataset

To rigorously test reasoning, we developed the MWP-MISTAKE dataset, built upon benchmarks like GSM-8K, MATH, MATHBENCH, and JEEBENCH. We designed two primary evaluation tasks:

  • Task-1 (T1): Detection & Rectification: The model identifies if a solution is correct. If incorrect, it must rectify the reasoning to find the true answer.
  • Task-2 (T2): Detection Only: The model identifies correctness without being explicitly asked to fix the intermediate steps.

Errors were injected using two strategies: Rule-Based Injection (e.g., swapping operators or deleting steps) and SLM-as-Bad-Reasoner, which uses smaller models like Llama-2-7b to generate realistic, flawed reasoning paths.

Figure 1: An example of the Detection and Rectification task (T1), comparing GPT-4o's successful correction against GPT-3.5 Turbo's failure to spot a calculation error.

Key Results and Performance Analysis

Our benchmarking reveals a substantial performance gap between frontier models and their smaller counterparts. GPT-4o emerged as the clear leader, showing a significant advantage in identifying and fixing mathematical blunders.

Figure 3: F1 scores for mistake detection across GPT-4o, GPT-4, and GPT-3.5 Turbo on multiple benchmarks.

Major Findings

  • GPT-4o Dominance: GPT-4o achieved an average F1 score of 0.87 across all datasets, outperforming GPT-4 by approximately 10% and smaller models by over 20%.
  • The "Memorization" Trap: Models frequently derived the correct final answer even when they failed to detect errors in the provided reasoning. This suggests they are often recalling answers from training data rather than logically verifying the prompt's steps.
  • The SLM Gap: Small Language Models (SLMs) like Phi-3 and Llama-2 performed poorly at mistake detection, often scoring below 0.40 F1, highlighting a massive reasoning divide.
  • Task Comparison: Performance generally improved in Task-1 compared to Task-2. Forcing a model to explicitly correct the logic (T1) appears to help it reach the correct final answer more reliably than simply asking for the answer despite the errors (T2).

Discussion and Conclusion

While GPT-4o shows promise, the struggle of most LLMs to detect reasoning errors—especially on complex datasets like JEEBENCH—indicates that mathematical reasoning remains an "Achilles' heel." Furthermore, "guided vs. general" instruction tests confirmed that models perform better when they recognize a specific dataset, pointing to persistent issues with data contamination.

To move forward, the field requires more "reasoning-dense" training data and evaluation processes that prioritize the logical path over the final result. Enhancing the self-correction capabilities of models is essential for creating robust, trustworthy AI systems for mathematical problem-solving.

Summary updated

Abstract

Large Language Models (LLMs) have been applied to Math Word Problems (MWPs) with transformative impacts, revolutionizing how these complex problems are approached and solved in various domains including educational settings. However, the evaluation of these models often prioritizes final accuracy, overlooking the crucial aspect of reasoning capabilities. This work addresses this gap by focusing on the ability of LLMs to detect and correct reasoning mistakes. We introduce a novel dataset MWP-MISTAKE, incorporating MWPs with both correct and incorrect reasoning steps generated through rule-based methods and smaller language models. Our comprehensive benchmarking reveals significant insights into the strengths and weaknesses of state-of-the-art models, such as GPT-4o, GPT-4, GPT-3.5Turbo, and others. We highlight GPT-$o's superior performance in mistake detection and rectification and the persistent challenges faced by smaller models. Additionally, we identify issues related to data contamination and memorization, impacting the reliability of LLMs in real-world applications. Our findings emphasize the importance of rigorous evaluation of reasoning processes and propose future directions to enhance the generalization and robustness of LLMs in mathematical problem-solving.
BibTeX
@misc{singh2024exposing,
  author = {Joykirat Singh and Akshay Nambi and Vibhav Vineet},
  title = {{Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning}},
  year = {2024},
  eprint = {2406.10834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10834},
}
Figure