TL;DR
This paper introduces Ape210K, a massive and diverse dataset of Chinese elementary school math word problems, significantly larger and richer in templates than existing benchmarks. They also propose a feature-enriched, copy-augmented seq2seq model that sets a new baseline for this challenging task, though it still falls short of human performance.
What Problem Does This Paper Address?
Automatic math word problem (MWP) solving is a crucial area in natural language understanding and artificial intelligence. Existing datasets for this task, like Math23K, are often too small and lack diversity in problem types and equation templates. This limits the ability of data-driven approaches, particularly deep learning models, to generalize well. The authors aim to address this by creating a much larger and more diverse dataset and developing a robust baseline model to push the state-of-the-art.
Methods
The core of their approach is a feature-enriched and copy-augmented sequence-to-sequence (seq2seq) model.
- Encoder-Decoder Architecture: They employ a standard encoder-decoder framework with soft attention, common in seq2seq tasks.
- Feature Enrichment: The encoder incorporates handcrafted features beyond just word embeddings. These include character information, part-of-speech tags, word identity, number identification, and the sorted index of numbers within the problem. These features are embedded and combined via a linear weighted sum.
- Pre-trained Embeddings: Word and character embeddings are pre-trained using word2vec on the problem description texts.
- Copying Mechanism: Inspired by its success in tasks like text summarization, they introduce a copying mechanism. This allows the model to directly copy tokens (numbers or symbols) from the input problem description to the output equation, which is particularly useful for handling numbers that appear in the problem but not in the general vocabulary.
- Equation Normalization: To handle variations in expressing equations (e.g., commutative property), they normalize the order of symbols in the extracted equations.
Datasets
- Ape210K: The primary contribution is this newly introduced dataset. It contains 210,488 Chinese elementary school math word problems, which is 9 times larger than Math23K. It also boasts 56,532 unique equation templates, 25 times more than Math23K. A significant portion (37.70%) of these templates involve "external constants" – numbers not explicitly present in the problem description but required for the solution, often relying on commonsense knowledge.
- Math23K: Used for comparison and evaluation. It's a widely used benchmark but significantly smaller and less diverse than Ape210K.
Key Insights and Ideas
- Scale and Diversity Matter: The sheer size and template diversity of Ape210K make it a significantly more challenging benchmark, highlighting the limitations of current models on real-world MWP tasks.
- Commonsense is Crucial: Many problems in Ape210K require commonsense knowledge (e.g., a chicken has 2 legs) to infer external constants, a capability that is difficult for current AI models.
- Copying Mechanism Boosts Performance: The copying mechanism proves effective, especially for problems with specific numbers that need to be directly transferred to the equation.
- Rich Features Help: Incorporating handcrafted features beyond basic embeddings improves model performance, suggesting that linguistic and structural cues are important.
Key Results
- On Math23K: The proposed feature-enriched, copy-augmented seq2seq model achieves 77.5% accuracy, outperforming the previous state-of-the-art by 3.2%.
- On Ape210K: The model achieves 70.20% accuracy on the test set, serving as a strong baseline. This is significantly higher than other models like StackDecoder (52.28%) and GTS-model (56.56%) on this dataset.
- Ablation Study: Removing input features (except character) drops accuracy by 2.9%, and removing the copying mechanism drops it by 2.72%, demonstrating the utility of these components. Removing all optimizations results in a significant drop to 66.22%.
- Error Analysis: The model performs well on common templates but struggles with problems involving long templates, external constants, and infrequent templates, indicating areas for future improvement.
Limitations and Future Work
- Long-Tail Problem: The model performs poorly on "long-tail" cases, such as problems with long templates, external constants, or infrequent templates. Addressing these challenging cases is identified as a key future direction.
- Human Performance Gap: Despite achieving a strong baseline, there remains a significant gap between the model's performance and human capabilities on Ape210K.
- Future Work: The authors suggest further research into handling more complex problem types, improving commonsense reasoning integration, and developing models that can better tackle the long-tail distribution of templates. They also leave problems with multiple unknown variables for future work.
Summary updated