Neural Math Word Problem Solver with Reinforcement Learning

Danqing Huang, Jing Liu, Chin-Yew Lin, Jian YinCOLING 2018 · OpenReview

Abstract: Danqing Huang, Jing Liu, Chin-Yew Lin, Jian Yin. Proceedings of the 27th International Conference on Computational Linguistics.

2018.

Introduction

Solving a math word problem is a multi-step cognitive feat that requires parsing natural language, identifying relevant quantities, determining the underlying mathematical structure, and executing calculations. Historically, AI systems for this task have relied on rigid rule-based logic or statistical models requiring heavy feature engineering. In "Neural Math Word Problem Solver with Reinforcement Learning," Huang et al. present a neural approach that leverages sequence-to-sequence (seq2seq) architectures while fixing the "hallucinations" and rigid training objectives that often plague standard models.

The Challenge: Hallucinations and Rigidity

Standard seq2seq models, which treat problem text as a source language and equations as a target language, suffer from two major "math-breaking" flaws: they generate "spurious numbers" (digits not present in the problem) and place correct numbers in incorrect positions within the equation.

Examples of baseline seq2seq model errors, including number hallucination and incorrect positioning.

Furthermore, traditional Maximum Likelihood Estimation (MLE) training is overly restrictive. It penalizes models for generating any equation string that doesn't exactly match the ground-truth label, even if the alternative equation is mathematically valid and yields the correct answer. This creates a "train-test discrepancy" where the model is penalized for valid logic simply because it doesn't match the training label word-for-word.

The CASS Model and Reinforcement Learning

To overcome these hurdles, the authors proposed CASS (Copy and Alignment Seq2Seq), which incorporates three key innovations:

  • Copy Mechanism: Instead of selecting numbers from a fixed vocabulary, the model learns to "point" to and copy numbers directly from the input text. This effectively eliminates the "spurious number" problem.
  • Alignment Mechanism: To fix the "wrong position" issue, a supervised attention mechanism provides the model with explicit information about which number in the text should align with which slot in the equation template during training.
  • Reinforcement Learning (RL): Using the REINFORCE algorithm, the model is trained to optimize for final answer accuracy rather than string similarity. By rewarding the model for correct numerical results (+1) and penalizing errors (-1), the system learns to explore diverse, valid mathematical strategies that MLE would otherwise suppress.

Experimental Evaluation

The researchers evaluated CASS across three major benchmarks: Alg514 (514 algebra problems), NumWord (2,871 problems), and the large-scale Dolphin18K dataset (over 18,000 problems from Yahoo! Answers). They also explored a Hybrid Ensemble approach, combining the neural model’s outputs as high-quality features for a traditional ranking model.

Results and Discussion

The results demonstrated that the combination of copying, alignment, and RL significantly outperforms standard seq2seq baselines, particularly on complex datasets.

Accuracy comparison of different model configurations across multiple benchmarks.

As shown in the experimental data, the jump from a baseline seq2seq model (19.7% on NumWordT6) to the full CASS model with RL (64.0%) is substantial. Notably, switching from MLE to RL training consistently improved performance, such as the increase from 13.1% to 15.9% on the challenging Dolphin18KT1 dataset. The hybrid model reached state-of-the-art performance, achieving 82.5% on Alg514 and 33.2% on Dolphin18KT6.

Conclusion

By treating numbers as unique entities and optimizing for solution accuracy through reinforcement learning, the CASS model represents a significant advancement in automated math problem solving. While challenges remain—such as incorporating commonsense knowledge (e.g., knowing there are 12 months in a year) and handling deep multi-step logic—this work proves that neural models can move beyond simple pattern matching toward genuine mathematical reasoning.

Summary updated

BibTeX
@inproceedings{huang2018neural,
  author = {Danqing Huang and Jing Liu and Chin-Yew Lin and Jian Yin},
  title = {{Neural Math Word Problem Solver with Reinforcement Learning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=NAS10I_Vg38},
}
Figure