A Meaning-based English Math Word Problem Solver with Understanding, Reasoning and Explanation

Chao-Chun Liang, Shih-Hong Tsai, Ting-Yun Chang, Yi-Chung Lin, Keh-Yih SuCOLING 2016

TL;DR

This paper introduces a novel meaning-based statistical framework for solving English math word problems (MWPs). By representing quantities with "role-tags" that capture their syntactic and semantic meaning, the system can better identify relevant information and explain its reasoning process in a human-comprehensible way. This approach aims to overcome limitations of purely rule-based or statistical methods in handling complex MWPs.

What Problem Does This Paper Address?

Solving math word problems (MWPs) is a challenging task in natural language understanding because it requires more than simple keyword matching. It demands understanding the relationships between quantities, performing logical inference, and generating a coherent explanation. Existing approaches have struggled: rule-based systems are difficult to build and maintain, while purely statistical methods can falter with complex problems or irrelevant information. This paper addresses the need for a more robust and interpretable MWP solver that can understand the nuances of language and provide clear reasoning.

Methods

The proposed MWP solver employs a pipelined architecture with several key modules:

  • Language Analyzer (LA): This module, likely using tools like Stanford CoreNLP, parses the input text to extract linguistic representations, including dependency trees and co-reference chains.
  • Solution Type Classifier (STC): An SVM classifier identifies the core mathematical operation needed (e.g., Addition, Subtraction, Multiplication, Division, Sum, Time-Variant-Quantity). It uses features like verb categories and keyword indicators.
  • Logic Form Converter (LFC): This module transforms the linguistic representations into formal logic forms (FOL predicates). A two-stage process is used: first, generating FOL predicates for sentence elements, and second, creating "quantity facts" (e.g., quan(q1,#,rose)=36) that capture the quantity, unit, object, and number. Crucially, "role-tags" (like nsubj, verb) are associated with these quantities to denote their syntactic and semantic roles.
  • Inference Engine (IE): This engine uses the logic forms and the identified solution type to perform inference and derive the answer. It utilizes a set of utilities for basic arithmetic operations and can prune irrelevant quantities based on their role-tags.
  • Explanation Generator (EG): This module takes the reasoning chain produced by the IE and generates a human-readable explanation of how the answer was obtained, often by converting the reasoning into an explanation tree.

The core innovation lies in the "role-tags" which provide a flexible way to annotate quantities with their physical meaning and context, enabling better operand identification and irrelevant information filtering.

Datasets

The system was evaluated on two publicly available datasets:

  • AI2-395: Contains 395 Addition and Subtraction MWPs, further divided into sub-datasets (MA1, IXL, MA2) with different feature categories.
  • IL-562: A collection of 562 arithmetic word problems, each solvable with a single math operation (Addition, Subtraction, Multiplication, or Division).

Key Insights and Ideas

  • Role-Tags for Meaning: The introduction of role-tags is a significant contribution. By explicitly linking syntactic roles (like subject, object) and semantic roles (like the verb associated with a quantity) to numerical values, the system gains a deeper understanding of the quantity's context. This is crucial for distinguishing relevant from irrelevant information.
  • Bridging Logic and Statistics: The paper proposes a hybrid approach that combines the interpretability of logic-based reasoning with the robustness of statistical methods. The statistical classifier (STC) identifies the problem type, while the logic engine (IE) performs the actual computation and reasoning, leveraging the semantically rich logic forms.
  • Human-Comprehensible Explanations: A key goal is to make the solver's reasoning transparent. The explanation generation module translates the internal reasoning steps into natural language, allowing users to understand why a particular answer was reached, which is vital for educational applications.
  • Pruning Irrelevant Information: The role-tagging mechanism allows the IE to effectively filter out quantities that do not fit the context of the question, a common challenge in MWPs. For instance, a quantity associated with a "drop" verb might be ignored if the question asks about quantities "picked."

Key Results

The paper reports strong performance on both AI2-395 and IL-562 datasets.

  • On AI2-395, the proposed system achieved an accuracy of 85.3% (total micro-average), significantly outperforming existing systems like UIUC (78.0%), ARIS (77.7%), and KAZB (64.0%).
  • On IL-562, the system achieved an accuracy of 79.5% (3-fold cross-validation), also surpassing UIUC (73.9%) and ARIS (73.7%).
  • The Solution Type Classifier (STC) demonstrated high accuracy, with rates ranging from 74.1% to 91.8% across different datasets.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Step Problems: The current solver is primarily designed for single-step MWPs. It cannot handle problems requiring multiple arithmetic operations (e.g., "Mary had 92 pieces of candy. She gave 4 pieces each to 9 friends. How many pieces of candy does Mary have left?").
  • Subtle Referring Relationships: The system struggles with more complex referring relationships, such as understanding that "customers" might refer to a combination of "women" and "men" in a given context.

Future work will focus on developing a goal-oriented approach to handle multi-step MWPs and enhancing the system's ability to resolve subtle referring relationships.

Summary updated

BibTeX
@inproceedings{liang2016meaning,
  author = {Chao-Chun Liang and Shih-Hong Tsai and Ting-Yun Chang and Yi-Chung Lin and Keh-Yih Su},
  title = {{A Meaning-based English Math Word Problem Solver with Understanding, Reasoning and Explanation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure