Designing a Tag-Based Statistical Math Word Problem Solver with Reasoning and Explanation

Chien-Tsung Huang, Yi-Chung Lin, Chao-Chun Liang, Kuang-Yi Hsu, Shen-Yun Miao, Wei-Yun Ma, Lun-Wei Ku, Churn-Jung Liau, Keh-Yih SuROCLING 2015 · OpenReview

TL;DR

This paper introduces a novel tag-based statistical framework for solving math word problems (MWPs) that aims to overcome limitations of existing rule-based and non-inferential statistical methods. By representing problem elements with tags and employing logic inference, their system achieves more robust understanding, precise answers, and generates explanations, paving the way for more interpretable AI problem solvers.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically solving math word problems (MWPs), a task that serves as a crucial testbed for "Machine Reading" (MR) capabilities. While "Big Data" focuses on correlations, understanding the underlying causality is key, and MR aims to enable machines to read and comprehend documents. Traditional MWP solvers fall into two main categories: rule-based approaches (with or without logic inference) and statistical approaches. Rule-based methods struggle with coverage (difficulty in creating comprehensive rules) and error accumulation. Statistical approaches, while better at handling irrelevant information, often lack logic inference, requiring custom handling for new problem types and hindering explanation generation. This paper seeks to bridge this gap by proposing a system that combines the robustness of statistical methods with the inferential power of logic, while also enabling explanation generation.

Methods

The proposed system is a tag-based statistical framework that integrates understanding and reasoning with logic inference. Key components include:

  • Language Analysis: This module processes each sentence of the MWP (body and question) and transforms it into a semantic representation tree.
  • Problem Resolution Module: This is the core of the system and comprises three sub-components:
  • Solution Type Classifier (TC): Determines the type of solution required for each question.
  • Logic Form Converter (LFC): Extracts relevant facts from the semantic representation tree and converts them into First Order Logic (FOL) predicates/functions. It also transforms questions into FOL-like utility functions.
  • Inference Engine (IE): Derives new facts from existing ones using inference rules and performs necessary mathematical operations.
  • Explanation Generation (EG): Generates a natural language explanation of how the answer was derived, based on the reasoning chain.

The "tag-based" aspect refers to the use of associated modifiers in the logic form (e.g., verb, agent, head, color) as tags to select appropriate information. This approach aims to be more robust to irrelevant information and provide more precise answers compared to purely statistical methods, while also offering the flexibility of rule-based systems without their coverage limitations.

Datasets

The paper mentions using an "elementary school math corpus" for training and evaluation. They have manually annotated 75 samples as a seed corpus, including syntactic trees, semantic trees, logic forms, and reasoning chains. The original corpus contains 23,493 problems across six grades.

Key Insights and Ideas

  • Bridging the Gap: The core insight is that a tag-based statistical framework can effectively combine the strengths of rule-based and statistical approaches for MWP solving, mitigating their respective weaknesses.
  • Robustness through Tagging: Using tags (modifiers in logic forms) makes the system more resilient to irrelevant information within the problem text.
  • Inferential Power: Incorporating logic inference allows for deeper understanding and more accurate problem-solving, going beyond simple pattern matching.
  • Explainability: The framework is designed to generate reasoning chains, which are crucial for producing human-understandable explanations, a significant step towards interpretable AI.
  • Flexibility: The tag-based logic representation allows for handling various question types with the same underlying problem logic.

Key Results

The paper reports on a prototype system that has been tested on a seed corpus. While specific quantitative results (like accuracy percentages) are not detailed in the provided text, the authors state that the "success of our pilot run has demonstrated the feasibility of the proposed approach." They also provide a detailed example of how their system processes a simple addition problem, correctly identifying quantities, operations, and generating the answer.

Limitations and Future Work

  • Limitations: The paper acknowledges that the coverage rate problem is a concern for rule-based approaches, and while their system aims to mitigate this, the extent of its coverage is not fully quantified in this extended abstract. The error accumulation problem is also mentioned as a challenge for rule-based systems.
  • Future Work: The authors plan to use the next few months to perform "weakly supervised learning" and "fine tune the system." They also mention that explanation generation (EG) details are omitted due to space limitations but refer to another publication for further information.

Summary updated

BibTeX
@inproceedings{huang2015designing,
  author = {Chien-Tsung Huang and Yi-Chung Lin and Chao-Chun Liang and Kuang-Yi Hsu and Shen-Yun Miao and Wei-Yun Ma and Lun-Wei Ku and Churn-Jung Liau and Keh-Yih Su},
  title = {{Designing a Tag-Based Statistical Math Word Problem Solver with Reasoning and Explanation}},
  booktitle = {ROCLING},
  year = {2015},
  url = {https://openreview.net/forum?id=z4up71JIiCv},
}
Figure