A Goal-Oriented Meaning-based Statistical Multi-Step Math Word Problem Solver with Understanding, Reasoning and Explanation

Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, Keh-Yih SuIJCAI 2017 · DOI

TL;DR

This paper introduces a novel goal-oriented, meaning-based statistical framework for solving multi-step math word problems. By transforming problems into logical forms with contextual role-tags and recursively decomposing goals, the system achieves human-like understanding and provides clear explanations for its reasoning.

What Problem Does This Paper Address?

Solving math word problems (MWPs) is a long-standing challenge in natural language understanding, requiring not just parsing text but also reasoning and calculation. While many approaches exist, they often struggle with multi-step problems, lack intermediate understanding, or fail to provide explanations. Previous methods either handled only single operations, had limitations in multi-step reasoning, or relied on complex rule-sets. This paper aims to bridge this gap by developing a system that can understand, reason through, and explain multi-step MWPs in a way that mimics human cognitive processes.

Methods

The proposed system employs a goal-oriented, meaning-based statistical approach. The core components include:

  • Language Analyzer (LA): Uses Stanford CoreNLP to obtain linguistic representations like dependency trees and co-reference chains.
  • Logic Form Converter (LFC): Transforms linguistic representations into first-order logic forms. Crucially, it associates quantities with "role-tags" (e.g., nsubj, verb, time) to capture their physical meaning and contextual relationships. This allows for a more flexible and intuitive representation than traditional methods. The LFC operates in a top-down, recursive manner, starting with the main goal and breaking it down into sub-goals.
  • Solution Type Classifier (STC): A statistical classifier that determines the appropriate operation (e.g., addition, subtraction) for a given goal.
  • Sub-goal Candidates Generator: Generates potential sub-goals based on rules.
  • Operands Identifier: A statistical model (SVM with linear kernel) that selects the correct operands from a set of known quantities and generated sub-goals. It uses features like math fact patterns, NP matching, and role-tag matching.
  • Inference Engine (IE): Executes the logic forms to compute the answer.
  • Explanation Generator (EG): Produces human-comprehensible explanations of the reasoning process, leveraging the generated logic forms and reasoning chain.

The key novelty lies in the explicit use of role-tags to represent the physical meaning of quantities, which informs both operation selection and operand identification, and the top-down, recursive decomposition of the problem's goal.

Datasets

The system is evaluated on two publicly available datasets:

  • IL-562: A collection of 562 MWPs, each involving a single arithmetic operation.
  • CC-600: A dataset of 600 MWPs designed to cover multi-step problems with four different arithmetic operations.

The paper compares its performance against existing systems like KAZB and the approach by Roy and Roth (2015) using n-fold cross-validation.

Key Insights and Ideas

  • Meaningful Representation: Associating quantities with role-tags (e.g., "Mary ate 3 apples" represented as quan(q1,#,apple)=3 & verb(q1,eat) & nsub(q1,Mary)) provides a rich, context-aware representation that goes beyond simple numerical values.
  • Goal Decomposition: A top-down, recursive decomposition of the main goal into sub-goals mirrors human problem-solving, making the process more interpretable.
  • Statistical Flexibility: Relying on statistical models for operation and operand selection allows the system to handle a wider range of problems than rule-based systems, without requiring extensive manual rule creation.
  • Explainability: The explicit use of meaning and the structured reasoning process naturally lead to human-comprehensible explanations, a significant advantage over "black box" solvers.

Key Results

The proposed system demonstrates significant improvements in accuracy compared to existing methods on both datasets.

  • IL-562: Achieved 80.1% accuracy.
  • CC-600: Achieved 53.5% accuracy.

These results are notably higher than those reported by Kushman et al. (2014) (73.7% on IL-562, 2.3% on CC-600) and Roy and Roth (2015) (73.9% on IL-562, 45.2% on CC-600). The authors attribute this improvement to the system's explicit checking of the physical meaning of selected quantities against the sub-goal's meaning.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations, but the performance on CC-600 (53.5%) suggests that multi-step reasoning and complex problem structures still present challenges. The reliance on statistical models means performance is tied to the quality and coverage of the training data.
  • Future Work: The authors suggest that their framework, which resembles human cognitive understanding, can be further extended. The availability of a web demo indicates a desire for broader adoption and testing. Future work could focus on improving performance on more complex problem types and exploring more sophisticated statistical models.

Summary updated

BibTeX
@inproceedings{liang2017goal,
  author = {Chao-Chun Liang and Yu-Shiang Wong and Yi-Chung Lin and Keh-Yih Su},
  title = {{A Goal-Oriented Meaning-based Statistical Multi-Step Math Word Problem Solver with Understanding, Reasoning and Explanation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/775},
}
Figure