Abstract
The creation of high-quality datasets to improve Large Language Model (LLM) reasoning remains a significant challenge, as current methods often suffer from generating low-quality/incorrect answers and limited information richness. We propose AgenticMath, a novel agentic method for generating high-quality mathematical question-answer pairs. Our method operates through four stages: (1) Seed Question Filtering, (2) Agentic Question Rephrasing via a multi-agent system, (3) Answer Augmentation using chain-of-thought reasoning, and (4) final Question and Answer Evaluation. Extensive experiments demonstrate that fine-tuning 3B-8B parameter LLMs on AgenticMath datasets (comprising only 30-60K samples) achieves competitive or superior performance on diverse benchmarks compared to baselines trained on much larger datasets (e.g., 400K or 2.3M samples). Our work demonstrates that targeted, high-quality data generation is a more efficient path to improving mathematical reasoning than large-scale, low-quality alternatives.
Introduction
While LLMs have shown impressive general reasoning, they still struggle with the precision and multi-step logic required for mathematics. To improve them, researchers typically use Supervised Fine-Tuning (SFT) on large synthetic datasets. However, synthetic data often suffers from a "quality ceiling": if the generated problems are unclear, logically inconsistent, or too simple, the model's reasoning performance plateaus regardless of data volume. AgenticMath addresses this by enforcing strict domain-specific constraints through an automated multi-agent framework.
Methodology
AgenticMath employs a rigorous four-stage pipeline to ensure quality control at every step of the data generation process:
- Seed Question Filter: Instead of using all human-authored questions as "seeds," the system uses an LLM to score them based on complexity, information value, and clarity. A $k$-nearest neighbor ($k$-NN) consistency check is used to smooth out noisy ratings.
- Agentic Problem Rephrase: A trio of agents (Rephrase, Review, and Revise) work in an iterative loop. The Rephrase agent creates variants of a seed question; the Review agent critiques them for solvability and logic; and the Revise agent fixes them. This ensures the final problem statement is challenging and logically sound.
- Answer Augment (Solution Generation): A solver agent generates detailed, step-by-step Chain-of-Thought (CoT) solutions to enhance numerical and logical correctness.
- Synthetic Data Evaluation: Final pairs are scored again, retaining only top-scoring samples. A "long-tail" diversity score is used to ensure broad coverage of mathematical concepts.
Experimental Setup
Problems were sourced from GSM8K (grade-school math) and MATH (competition-level math). The resulting AgenticMathQA dataset was released in 30K, 60K, and 90K versions. Models were evaluated across six benchmarks, including in-domain (GSM8K, MATH) and out-of-domain (CollegeMath, DeepMind Mathematics, OlympiadBench, and TheoremQA) tasks.
Key Results and Insights
The "Less Is More" Principle
The most striking finding is that data quality is a more powerful lever than data scale. Models trained on just 5%–15% of the data used by previous state-of-the-art methods achieved superior results. For example, an AgenticMath-Qwen2.5-3B model trained on 30K samples achieved an average score of 53.7, significantly outperforming baselines like MathFusion (38.1) and DART-Math (47.4) that used similar or larger data amounts.
Performance and Generalization
As shown in the performance table below, AgenticMath consistently improves performance across various base models. The models showed significant gains in Out-of-Domain (OOD) tasks, suggesting that high-quality synthetic data helps models learn fundamental reasoning patterns rather than just memorizing specific problem types.
Agentic Iteration vs. One-Shot Generation
Using a Review-Revise loop significantly reduces hallucinations and logical inconsistencies compared to simple one-shot generation. Furthermore, AgenticMath posits that the problem statement itself determines the upper bound of reasoning quality; if a question is poorly posed, the reasoning chain is inherently limited.
Conclusion and Future Work
AgenticMath demonstrates that high-quality, agent-refined data is more effective for fine-tuning than massive, low-quality datasets. Future work will explore extending this agentic approach to multimodal settings (e.g., geometric figures) and investigating whether these "quality over quantity" trends hold at the scale of millions of samples. Currently, the framework relies on teacher models like GPT-4o-mini, and moving toward fully open-source agentic roles is a key next step.
Summary updated