The Most Uncreative Examinee: A First Step toward Wide Coverage Natural Language Math Problem Solving

Takuya Matsuzaki, Hidenao Iwane, Hirokazu Anai, Noriko H. AraiAAAI 2014 · DOI

Abstract

Automated math problem solving has traditionally been limited to narrow domains or simple arithmetic. This paper introduces a prototype system capable of addressing the "wide coverage" mathematics found in university entrance exams. The system utilizes a multi-stage pipeline that bridges natural language processing with formal mathematical reasoning by translating problems into Zermelo-Fraenkel (ZF) set theory and subsequently simplifying them into decidable sub-theories. Evaluation on mock entrance exams demonstrates that this symbolic approach can match or exceed average human performance in several categories.

Introduction

While artificial intelligence has made significant progress in solving arithmetic word problems, university-level mathematics remains a formidable frontier. These problems present two primary challenges: linguistic complexity—requiring the resolution of coreference and multi-sentence logic—and mathematical diversity, spanning geometry, calculus, and linear algebra. Most existing solvers are "brittle," specialized for single problem types. This research proposes a unified pipeline to handle a broad spectrum of mathematical topics by connecting formal linguistics with automated reasoning.

Methodology: The Solving Pipeline

The authors propose a multi-stage architecture to transform natural language into solvable mathematical constraints.

The multi-stage problem-solving pipeline, from annotated text to inference.
  1. Semantic Parsing: The system employs Combinatory Categorial Grammar (CCG) to convert sentences into logical forms. To maintain context across sentences, such as variable bindings, it uses Dynamic Predicate Logic (DPL).
  2. The ZF Bridge: A key innovation is the use of Zermelo-Fraenkel (ZF) set theory as a universal intermediate representation. ZF is expressive enough to capture nearly all undergraduate mathematics, serving as a "common language" before specific solving strategies are applied.
  3. Formula Rewriting: Because ZF is generally undecidable, a "greedy" rewriting engine applies equivalence-preserving rules to simplify ZF formulas. The goal is to reduce them into decidable sub-theories like Real Closed Field (RCF) or Peano Arithmetic (PA).
  4. Automated Reasoning: Once simplified, problems are passed to specialized solvers. For RCF problems, the system uses Quantifier Elimination (QE), a powerful technique that can find all possible values for a variable given a set of polynomial constraints.

Evaluation and Results

The system was evaluated on two primary benchmarks: 99 problems from top national national universities in Japan and real-world mock exams provided by the Yoyogi Seminar, taken by thousands of human students.

On problems categorized under Real Closed Field (primarily algebra and geometry), the system solved 60% with manual translations and 50% using semi-automatic linguistic processing. Crucially, the system's performance was compared directly against human test-takers.

Performance comparison on Science Course mock tests against human averages.

In four out of six mock test sets (across Humanities and Science courses), the system’s scores matched or exceeded the human average. For instance, in the July 2013 Science mock test, the system scored 40/120, significantly higher than the human average of 21.8.

Discussion and Limitations

The system's "uncreative" nature—relying on rigorous symbolic logic rather than human intuition—is both a strength and a limitation. While it excels at perfect logical execution, it faces a computational bottleneck with RCF-QE, which is doubly exponential. The success of the solver depends heavily on the linguistic engine's ability to produce "clean" formulas that avoid computational explosion.

Furthermore, the system currently struggles with transcendental functions (e.g., sin, log) and lacks a backtracking mechanism in its rewriting engine. Future work will focus on automating coreference resolution, expanding into probability and combinatorics, and improving search strategies for formula simplification.

Conclusion

This research represents a significant step toward wide-coverage natural language math solving. By using ZF set theory as a bridge, the system demonstrates that a unified symbolic pipeline can tackle the diverse and linguistically complex problems found in higher education entrance exams, often outperforming the average human examinee.

Summary updated

BibTeX
@inproceedings{matsuzaki2014most,
  author = {Takuya Matsuzaki and Hidenao Iwane and Hirokazu Anai and Noriko H. Arai},
  title = {{The Most Uncreative Examinee: A First Step toward Wide Coverage Natural Language Math Problem Solving}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.8869},
}
Figure