Math Word Problem Solving by Generating Linguistic Variants of Problem Statements

Syed Rifat Raiyan, Md Nafis Faiyaz, Shah Md. Jawad Kabir, Mohsinul Kabir, Hasan Mahmud, Md Kamrul HasanACL 2023

TL;DR

This paper proposes a novel framework for math word problem (MWP) solving that enhances model robustness and mathematical reasoning by generating paraphrased linguistic variants of problem statements. By training on these diverse variants and employing a majority voting mechanism on candidate predictions, the authors achieve improved accuracy and a more reliable MWP solver.

What Problem Does This Paper Address?

Solving math word problems (MWPs) is a long-standing challenge in Artificial General Intelligence (AGI). Existing MWP solvers often rely on shallow heuristics or spurious correlations, making them susceptible to minor changes in problem phrasing. This fragility hinders their ability to generalize and perform robust mathematical reasoning. The core challenge is to build models that can truly understand the underlying mathematical relationships within a problem, regardless of how it's worded, and consistently derive the correct solution.

Methods

The proposed framework tackles MWP solving through a two-pronged approach:

  1. Linguistic Variant Generation: The authors leverage OpenAI's GPT-3 models (text-davinci-003 and gpt-3.5-turbo) to generate multiple linguistic variants of an original MWP. These variants include paraphrased structures, altered object/entity names, and the addition of irrelevant information. This process aims to expose the model to a wider range of linguistic expressions for the same underlying mathematical problem.
  2. DeBERTa-based Solver with Majority Voting: Each of the generated variants, along with the original problem, is fed into a DeBERTa-based MWP solver. DeBERTa, with its disentangled attention mechanism and enhanced mask decoder, is chosen for its strong performance in natural language processing tasks. Crucially, the model generates multiple candidate solution expressions for each input. A majority voting mechanism is then applied to these candidate predictions (after simplification) to select the most frequent one as the final answer. This voting step is performed during testing and inference, not training.

Datasets

  • MAWPS (Math Word Problem Solving): A benchmark dataset used for training and evaluation.
  • SVAMP (Selective Math Word Problems): A challenge dataset used for testing robustness.
  • PARAMAWPS (Paraphrased Math Word Problem Solving Repository): A newly introduced dataset created by the authors. It consists of 16,278 single-equation MWPs, generated by paraphrasing, adding adversarial information, and inverting questions from the MAWPS dataset. This dataset is designed to be more challenging and diverse than existing benchmarks.

Key Insights and Ideas

  • Linguistic Diversity Enhances Reasoning: Training on paraphrased and varied problem statements significantly improves a model's mathematical reasoning capabilities, making it more robust to phrasing changes.
  • Majority Voting as a Robustness Mechanism: Aggregating predictions from multiple linguistic variants through majority voting helps to filter out incorrect solutions and increases the overall reliability of the MWP solver.
  • DeBERTa's Suitability for MWP: The DeBERTa model, with its advanced attention mechanisms, proves to be a strong backbone for MWP solving, especially when augmented with the proposed variant generation and voting strategy.
  • LLMs as Paraphrasers, Not Solvers: While LLMs like GPT-3 are effective at generating diverse problem variants, their direct application to solving complex MWPs (in a zero-shot setting) shows significantly lower accuracy compared to fine-tuned smaller models like DeBERTa.

Key Results

  • The proposed DeBERTa model with paraphrasing and voting achieved an accuracy of 91.0% on the MAWPS dataset, outperforming baseline models.
  • On the newly introduced PARAMAWPS dataset, the model achieved 74.1% accuracy with the vanilla DeBERTa and 79.1% with the voting mechanism, demonstrating its effectiveness on a more challenging benchmark.
  • The model achieved 63.5% accuracy on the SVAMP challenge dataset, which is notably higher than other baseline models.
  • Ablation studies showed that the paraphrasing model and voting mechanism contributed to accuracy improvements. Specifically, the voting mechanism improved accuracy by 5.04% on PARAMAWPS.
  • Experiments with LLMs (GPT-J, GPT-3 variants) in a zero-shot setting showed significantly lower performance (e.g., 73.0% for gpt-3.5-turbo on PARAMAWPS) compared to the fine-tuned DeBERTa model.

Limitations and Future Work

  • Temporal Overhead: Generating linguistic variants using GPT-3 introduces a temporal overhead, which might make the architecture less suitable for real-time applications.
  • Tie-breaking in Voting: The current majority voting module lacks a proper tie-breaking strategy, which could be improved with weighted votes based on semantic similarity.
  • Future Work: The authors plan to:
  • Incorporate a Tree-based decoder.
  • Introduce weighted voting for improved tie-breaking.
  • Experiment with harder problem text variations (e.g., grammatical errors).
  • Conduct thorough error analysis to identify model weaknesses.
  • Expand research to multi-equation, multi-step deduction, and domain-knowledge problems.

Summary updated

BibTeX
@inproceedings{raiyan2023math,
  author = {Syed Rifat Raiyan and Md Nafis Faiyaz and Shah Md. Jawad Kabir and Mohsinul Kabir and Hasan Mahmud and Md Kamrul Hasan},
  title = {{Math Word Problem Solving by Generating Linguistic Variants of Problem Statements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure