TL;DR
This paper demonstrates that strategically diversifying preference data—rather than just tweaking alignment algorithms—is the key to boosting LLM mathematical reasoning. The authors introduce Diversified-ThinkSolve (DTS), a lightweight method that systematically decomposes math problems into diverse strategic pathways, achieving up to a 7.1% performance gain on GSM8K with virtually zero computational overhead (1.03×).
What Problem Does This Paper Address?
While preference optimization techniques like Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO) have successfully aligned Large Language Models (LLMs) with human preferences on general tasks, mathematical reasoning remains a stubborn bottleneck.
Standard preference datasets typically rely on random temperature sampling or single-path Chain-of-Thought (CoT) prompting. These approaches often produce surface-level variations or converge prematurely on a single solution path, failing to expose the model to the rich, diverse problem-solving strategies required to master complex mathematics. The authors set out to explore how structured data generation can create more effective preference data specifically tailored for mathematical alignment.
Methods
The authors investigate how different data generation strategies impact preference learning for math. They compare standard temperature sampling, Chain-of-Thought (CoT) prompting, and Monte Carlo Tree Search (MCTS) against their novel approach: Diversified-ThinkSolve (DTS).
DTS leverages the DSPy framework to explicitly decouple problem-solving into two distinct phases:
- Thought Generation: A
ThoughtGeneratormodule prompts the model to propose 5 distinct high-level strategic approaches or perspectives for a given problem (e.g., identifying relevant formulas, techniques, or alternative angles). - Solution Execution: A
SolutionGeneratormodule takes each unique approach and executes a full, detailed step-by-step solution.
To construct the preference pairs, they apply a "mixed correctness" filter, selecting problems where 2–3 out of the 5 generated solutions are correct. They use a top-tier reward model (Nvidia's Llama-3.1-Nemotron-70B-Reward-HF) to select the highest-scoring correct solution as the preferred response ($y_w$) and the highest-scoring incorrect solution as the dispreferred response ($y_l$). These pairs are then used to train the model via SFT, DPO, ORPO, or SimPO.
Datasets
- Training/Data Generation: A deduplicated subset of the MetaMathQA dataset, containing 13,929 unique mathematical queries and solutions derived from the GSM8K and MATH training sets.
- Evaluation: The standard GSM8K test set (1,319 problems) for 0-shot and 5-shot accuracy, and the MATH-500 test subset.
Key Insights and Ideas
- Strategic Diversity Beats Surface Variation: Simply generating multiple answers via temperature sampling doesn't provide a rich enough learning signal. Forcing the model to explore fundamentally different mathematical pathways (e.g., using inverse operations vs. direct algebra) creates far more informative preference pairs.
- Data Quality > Algorithm Choice: The performance gap between different data generation strategies was significantly larger than the gap between alignment algorithms (DPO vs. SimPO). Investing in better data curation yields higher returns than algorithmic tweaking.
- Heavy Search is Inefficient for Alignment Data: While MCTS systematically explores solution trees, it is computationally expensive (4.85× compute) and actually resulted in worse and more unstable alignment performance compared to the lightweight DTS approach.
- Math Demands Gentle Updates: Through hyperparameter tuning, the authors found that mathematical alignment requires significantly lower learning rates (e.g., 7e-7 for ORPO) than standard instruction-following tasks to avoid disrupting the model's adherence to strict symbolic and formal rules.
Key Results
- Performance Gains: Using Llama-3.1-8B-Instruct as the base model, DTS combined with SimPO achieved 83.2% on GSM8K (a 7.1% absolute improvement). Combined with DPO, it achieved 52.4% on MATH (a 4.2% absolute improvement).
- Computational Efficiency: DTS achieved these top-tier results with a mere 1.03× relative compute overhead compared to baseline sampling. In contrast, CoT required 1.99× compute, and MCTS required a massive 4.85× compute while underperforming.
- Training Stability: DTS demonstrated remarkably stable performance progression across training epochs on GSM8K, whereas CoT and MCTS suffered from severe training instability and performance degradation in later epochs.
Limitations and Future Work
- Benchmark Scope: The evaluation is restricted to standard benchmarks (GSM8K and MATH), which may not capture the full breadth of real-world or advanced mathematical domains.
- Reward Model Dependency: The pipeline relies heavily on automated reward models to score solutions. Even the best model used (Nemotron-70B) has a ~3% error rate, which can introduce subtle biases or incorrect preference pairs into the training data.
- Model Scale: The study only evaluates an 8B parameter model; it remains to be seen how these data diversification strategies scale to much larger or smaller architectures.
Summary updated