TL;DR
This paper introduces a novel tag-based statistical approach for solving English math word problems (MWPs). By annotating quantities with rich syntactic and semantic information, their system achieves better understanding and reasoning, outperforming previous methods and handling a wider range of problem types beyond simple addition and subtraction.
What Problem Does This Paper Address?
Solving math word problems (MWPs) is a long-standing challenge in Natural Language Understanding (NLU) because it requires more than just keyword matching. It demands true comprehension, reasoning, and the ability to extract relevant information while ignoring distractors. Existing approaches often fall into a few categories: rule-based systems struggle with coverage and ambiguity, and many are limited to basic arithmetic operations. Purely statistical methods, while more flexible, can be overly sensitive to irrelevant information and may not perform well on complex problems without deep understanding. This paper aims to bridge this gap by developing a system that can understand, reason about, and explain MWPs, handling a broader spectrum of problem types.
Methods
The proposed system employs a tag-based statistical framework with several key modules:
- Language Analysis (LA): Utilizes the Stanford CoreNLP suite to perform tokenization, sentence splitting, POS tagging, lemmatization, named entity recognition, parsing, and co-reference resolution. This generates linguistic information, particularly dependency relations and co-reference chains, crucial for understanding relationships between entities.
- Solution Type Classifier (STC): A SVM classifier trained to predict the mathematical operation (e.g., addition, subtraction, multiplication, division, comparison, algebra) needed to solve the problem based on global information across sentences.
- Logic Form Converter (LFC): Transforms the linguistic information into first-order logic (FOL) predicates. A two-stage process is used: first, generating domain-independent syntactic predicates, and second, creating domain-dependent logic forms. Crucially, it introduces "tags" – associated syntactic and semantic information linked to quantities – to help identify relevant operands and filter out irrelevant ones. This is a core innovation, making the system more robust to noise.
- Inference Engine (IE): Derives new facts using inference rules and performs mathematical operations on selected facts to arrive at the answer. It leverages the tags generated by the LFC to prune irrelevant quantities.
- Explanation Generator (EG): Based on the reasoning chain from the IE, it generates a natural language explanation of how the answer was derived, using pre-defined templates for different operations.
Datasets
The experiments were conducted on datasets provided by Hosseini et al. (2014):
- MA1: Covers simple MWPs on addition and subtraction for third, fourth, and fifth graders.
- MA2: Includes more irrelevant information compared to MA1.
- IXL: Features more information gaps.
These datasets are noted as the first publically available benchmarks for comparing MWP systems.
Key Insights and Ideas
- The Power of "Tags": The central idea is annotating quantities with syntactic and semantic "tags." This allows the system to precisely identify what information is relevant to the question, significantly improving robustness against irrelevant details.
- Bridging Rule-Based and Statistical: The approach combines the structured reasoning of logic forms with the flexibility and learning capabilities of statistical classifiers (SVM for STC).
- Beyond Simple Arithmetic: The tag-based approach and the STC's ability to classify various solution types enable the system to handle a much wider range of math operations, including multiplication, division, comparison, and algebra, unlike many prior systems limited to addition and subtraction.
- Explainability Matters: The inclusion of an Explanation Generator is a significant feature, providing transparency into the system's reasoning process, which is crucial for user trust and debugging.
Key Results
The paper reports strong performance on the MA1, MA2, and IXL datasets. In a 3-fold cross-validation setting, their system achieved an overall accuracy of 84.8%. This significantly outperformed ARIS (77.7%) and KAZB (64.0%). For the "Gold Solution Type" evaluation, their system achieved an impressive 97.5% accuracy.
Limitations and Future Work
- Limitations: The paper acknowledges that the coverage rate of rules can be a problem for rule-based systems, and while their tag-based approach mitigates this, it's an ongoing challenge. The system's sensitivity to irrelevant information is reduced but not entirely eliminated. The current system primarily handles English MWPs.
- Future Work: The authors suggest that their tag-based framework can be extended to handle more complex problem types and potentially other languages. Further research could explore more sophisticated inference mechanisms and ways to further improve robustness against highly complex or ambiguous problem formulations.
Summary updated