Abstract
Large Language Models (LLMs) often struggle with the rigorous symbolic reasoning required for mathematics, frequently failing to generalize beyond simple forward-reasoning patterns. We present MetaMath, a fine-tuning framework that enhances mathematical reasoning by "bootstrapping" existing datasets. By rewriting problems from multiple perspectives—including rephrasing and reverse-solving—we created the MetaMathQA dataset. Our results show that MetaMath-70B outperforms GPT-3.5-Turbo on grade-school math benchmarks, demonstrating the power of data diversity in teaching models "meta-knowledge" of mathematics.
Introduction
While LLMs have revolutionized natural language processing, their performance on multi-step mathematical reasoning remains a significant challenge. Open-source models like LLaMA-2 often lag behind closed-source counterparts like GPT-4. The bottleneck is frequently attributed to data quality and diversity; standard fine-tuning typically provides a single "Question $\rightarrow$ Answer" view, which can lead to pattern memorization rather than true understanding. Furthermore, LLMs suffer from a "reversal curse," where they can reason from cause to effect but fail to reverse the logic.
Methods: Question Bootstrapping
The core innovation of MetaMath is Question Bootstrapping. This method transforms a single mathematical question into multiple training samples using three primary strategies:
- Answer Augmentation (AnsAug): Generating diverse reasoning paths for the same question and using rejection sampling to keep only correct paths.
- Question Rephrasing: Rewriting the question text to maintain logical identity while varying linguistic expression.
- Backward Reasoning: Forcing the model to reason from the answer back to the variables through Self-Verification (SV) and FOBAR (Fill-in-the-blank-with-Answer-Reasoning).
By combining these, the model learns mathematical relationships from multiple angles, effectively performing multi-view data augmentation.
The MetaMathQA Dataset
Using these bootstrapping techniques, we introduced MetaMathQA, a dataset containing 395,000 samples derived from GSM8K and MATH. MetaMathQA exhibits significantly higher diversity than the original datasets, as measured by embedding distance metrics. Analysis shows that while adding more forward reasoning samples eventually leads to performance saturation, the inclusion of bootstrapped questions provides steady gains in accuracy and robustness.
Experimental Results
MetaMath models achieved state-of-the-art results for open-source models of their respective sizes. Notably, MetaMath-70B achieved an accuracy of 82.3% on GSM8K, surpassing GPT-3.5-Turbo (80.8%).
Key Insights
- Breaking the Reversal Curse: MetaMath models show massive improvements in "backward" tasks, suggesting a more robust internal representation of mathematical logic.
- Lower Perplexity: MetaMathQA has lower perplexity for the models compared to standard Chain-of-Thought (CoT) datasets, indicating it is a more effective activator for latent knowledge.
- Out-of-Distribution (OOD) Robustness: MetaMath outperformed other math-tuned models on the DROP dataset, proving that bootstrapping teaches generalizable reasoning rather than benchmark-specific tricks.
Conclusion and Future Work
MetaMath demonstrates that data diversity and multi-view reasoning are critical for improving LLM performance in mathematics. While performance still degrades with question length, the success of MetaMath-70B suggests that closing the gap with models like GPT-4 is possible through sophisticated data augmentation. Future work will focus on extending these techniques to even more complex reasoning chains and larger-scale architectures.
Summary updated