MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models

Weihao You, Shuo Yin, Xudong Zhao, Zhilong Ji, Guoqiang Zhong, Jinfeng BaiNAACL 2024 · DOI

Sure, here's a blog-style summary of the paper "MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models":

TL;DR

This paper introduces MuMath, a novel multi-perspective data augmentation technique for improving the mathematical reasoning capabilities of large language models (LLMs). By combining several new augmentation methods, MuMath significantly boosts the performance of open-source LLMs on challenging math benchmarks, achieving state-of-the-art results among tool-free methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive capabilities across various NLP tasks, including mathematical reasoning. However, there's a persistent gap between proprietary LLMs (like GPT-4) and open-source LLMs in solving complex mathematical problems. Existing approaches to bridge this gap fall into two main categories: tool-use methods that leverage external interpreters (which lack transparency in their reasoning process) and tool-free methods that display the reasoning process but often suffer from lower accuracy. Tool-free methods typically rely on limited augmentation techniques like rephrasing or difficulty enhancement. This paper aims to improve the accuracy and transparency of tool-free LLMs for mathematical reasoning by developing a more comprehensive and effective data augmentation strategy.

Methods

The core contribution of this paper is the MuMath dataset and the associated multi-perspective data augmentation methods. These methods aim to enhance mathematical reasoning by:

  • Data Reformulation:
    • Reorganization: Rewriting solutions to be more detailed and easier to understand, breaking down complex steps into explicit instructions.
  • Rephrasing: Rewriting questions and answers while preserving their original meaning, similar to existing methods but integrated into a broader framework.
  • Backward Creation:
    • Backward-Forward Transformation (BF-Trans): A novel approach that transforms forward reasoning problems into arithmetic problems that can be solved more directly, essentially reversing the problem-solving direction. This is combined with the FOBAR method.
  • Question Alteration:
    • Expression Replacement: A new method that alters mathematical expressions within solutions to create new, logically sound questions with different calculation paths. This offers a more controlled way to introduce variation compared to arbitrary difficulty increases.
  • Difficulty Enhancement: Increasing the complexity of existing questions by adding constraints or modifying context, building on prior work.
  • Nested Multi-task Construction: Integrating auxiliary tasks like question outlining and solution planning directly into the main math problem-solving task within a single sample. This fine-grained approach aims to improve the model's ability to manage complex reasoning processes.

The authors also employed Majority Sampling Finetuning (MSF) to improve answer accuracy by selecting the most frequent answer from multiple generated solutions.

Datasets

The paper uses two widely recognized mathematical reasoning benchmarks for training and evaluation:

  • GSM8K: A dataset of high-quality elementary school math problems.
  • MATH: A more challenging dataset encompassing seven subjects from prealgebra to precalculus, often used for math competitions.

The authors created the MuMath dataset by applying their proposed augmentation methods to these existing datasets. The final MuMath dataset comprises approximately 304K samples.

Key Insights and Ideas

  • Multi-perspective augmentation is key: Simply rephrasing or increasing difficulty isn't enough. Combining diverse augmentation strategies (reformulation, backward creation, question alteration, and nested multi-task learning) leads to significantly better performance.
  • Reorganization enhances understandability: Detailed, step-by-step solutions, even if longer, can improve a model's ability to grasp problem-solving processes.
  • BF-Trans offers a novel reasoning path: Transforming problems into arithmetic ones provides a different angle for LLMs to tackle mathematical challenges.
  • Expression replacement is a controlled way to vary logic: This method allows for introducing new calculation logic without arbitrary difficulty spikes, making it a valuable tool for data augmentation.
  • Nested multi-task learning boosts reasoning: Integrating auxiliary tasks directly into the main task sample helps LLMs develop more robust reasoning skills.
  • MSF improves answer accuracy: Forcing models to consider multiple potential solutions and select the most common one can lead to more reliable answers.

Key Results

  • The MuMath-70B model achieved state-of-the-art performance among tool-free methods, reaching 88.3% on GSM8K and 34.5% on MATH.
  • Compared to the baseline LLaMA-2 SFT, MuMath showed significant accuracy increases of 34.6% on GSM8K and 18.9% on MATH.
  • MuMath also surpassed existing tool-free methods like MetaMath and MuggleMath across various model sizes (7B, 13B, and 70B).
  • Ablation studies demonstrated that each augmentation method individually contributes to performance improvement, and their combination is synergistic.
  • The paper also showed that MuMath models performed well on out-of-domain datasets like SVAMP, MAWPS, and ASDiv, sometimes even surpassing tool-use LLMs like ToRA.

Limitations and Future Work

  • Data Size: The MuMath dataset (around 304K samples) is smaller than some existing augmented datasets (e.g., MetaMathQA at 395K), yet it achieves better performance, highlighting the effectiveness of the augmentation strategies.
  • Out-of-Domain Reasoning: While MuMath improves in-domain reasoning, the paper notes that GSM8K augmented data doesn't significantly boost MATH performance, and vice-versa. This suggests that domain-specific augmentation might be necessary for broader generalization.
  • Future Work: The authors plan to explore other augmentation methods to further enhance mathematical reasoning for tool-free LLMs and investigate more auxiliary tasks for nested multi-task learning.

Summary updated

BibTeX
@inproceedings{you2024mumath,
  author = {Weihao You and Shuo Yin and Xudong Zhao and Zhilong Ji and Guoqiang Zhong and Jinfeng Bai},
  title = {{MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.185},
}
Figure