Data Diversification Methods In Alignment Enhance Math Performance In LLMs

Berkan Dokmeci, Qingyang Wu, Ben Athiwaratkun, Ce Zhang, Shuaiwen Leon Song, James Zou

TL;DR

This paper demonstrates that strategically diversifying preference data—rather than just tweaking alignment algorithms—is the key to boosting LLM mathematical reasoning. The authors introduce Diversified-ThinkSolve (DTS), a lightweight method that systematically decomposes math problems into diverse strategic pathways, achieving up to a 7.1% performance gain on GSM8K with virtually zero computational overhead (1.03×).

What Problem Does This Paper Address?

While preference optimization techniques like Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO) have successfully aligned Large Language Models (LLMs) with human preferences on general tasks, mathematical reasoning remains a stubborn bottleneck.

Standard preference datasets typically rely on random temperature sampling or single-path Chain-of-Thought (CoT) prompting. These approaches often produce surface-level variations or converge prematurely on a single solution path, failing to expose the model to the rich, diverse problem-solving strategies required to master complex mathematics. The authors set out to explore how structured data generation can create more effective preference data specifically tailored for mathematical alignment.

Methods

The authors investigate how different data generation strategies impact preference learning for math. They compare standard temperature sampling, Chain-of-Thought (CoT) prompting, and Monte Carlo Tree Search (MCTS) against their novel approach: Diversified-ThinkSolve (DTS).

DTS leverages the DSPy framework to explicitly decouple problem-solving into two distinct phases:

  1. Thought Generation: A ThoughtGenerator module prompts the model to propose 5 distinct high-level strategic approaches or perspectives for a given problem (e.g., identifying relevant formulas, techniques, or alternative angles).
  2. Solution Execution: A SolutionGenerator module takes each unique approach and executes a full, detailed step-by-step solution.
Figure 1: Diversified-ThinkSolve (DTS) modular rea- soning pipeline for generating diverse mathematical problem solutions. Each math problem is first processed by a ThoughtGenerator to propose multiple solution approaches. Then utilizing the SolutionGenerator with each approach, we are given multiple solutions, con- tributing to a diverse set of preference data.

To construct the preference pairs, they apply a "mixed correctness" filter, selecting problems where 2–3 out of the 5 generated solutions are correct. They use a top-tier reward model (Nvidia's Llama-3.1-Nemotron-70B-Reward-HF) to select the highest-scoring correct solution as the preferred response ($y_w$) and the highest-scoring incorrect solution as the dispreferred response ($y_l$). These pairs are then used to train the model via SFT, DPO, ORPO, or SimPO.

Datasets

  • Training/Data Generation: A deduplicated subset of the MetaMathQA dataset, containing 13,929 unique mathematical queries and solutions derived from the GSM8K and MATH training sets.
  • Evaluation: The standard GSM8K test set (1,319 problems) for 0-shot and 5-shot accuracy, and the MATH-500 test subset.

Key Insights and Ideas

  • Strategic Diversity Beats Surface Variation: Simply generating multiple answers via temperature sampling doesn't provide a rich enough learning signal. Forcing the model to explore fundamentally different mathematical pathways (e.g., using inverse operations vs. direct algebra) creates far more informative preference pairs.
  • Data Quality > Algorithm Choice: The performance gap between different data generation strategies was significantly larger than the gap between alignment algorithms (DPO vs. SimPO). Investing in better data curation yields higher returns than algorithmic tweaking.
  • Heavy Search is Inefficient for Alignment Data: While MCTS systematically explores solution trees, it is computationally expensive (4.85× compute) and actually resulted in worse and more unstable alignment performance compared to the lightweight DTS approach.
  • Math Demands Gentle Updates: Through hyperparameter tuning, the authors found that mathematical alignment requires significantly lower learning rates (e.g., 7e-7 for ORPO) than standard instruction-following tasks to avoid disrupting the model's adherence to strict symbolic and formal rules.

Key Results

  • Performance Gains: Using Llama-3.1-8B-Instruct as the base model, DTS combined with SimPO achieved 83.2% on GSM8K (a 7.1% absolute improvement). Combined with DPO, it achieved 52.4% on MATH (a 4.2% absolute improvement).
  • Computational Efficiency: DTS achieved these top-tier results with a mere 1.03× relative compute overhead compared to baseline sampling. In contrast, CoT required 1.99× compute, and MCTS required a massive 4.85× compute while underperforming.
  • Training Stability: DTS demonstrated remarkably stable performance progression across training epochs on GSM8K, whereas CoT and MCTS suffered from severe training instability and performance degradation in later epochs.
Figure 3: Performance progression across training epochs for different data generation strategies using optimal hyperparameters.

Limitations and Future Work

  • Benchmark Scope: The evaluation is restricted to standard benchmarks (GSM8K and MATH), which may not capture the full breadth of real-world or advanced mathematical domains.
  • Reward Model Dependency: The pipeline relies heavily on automated reward models to score solutions. Even the best model used (Nemotron-70B) has a ~3% error rate, which can introduce subtle biases or incorrect preference pairs into the training data.
  • Model Scale: The study only evaluates an 8B parameter model; it remains to be seen how these data diversification strategies scale to much larger or smaller architectures.

Summary updated

Abstract

While recent advances in preference learning have enhanced alignment in human feedback, mathematical reasoning remains a persistent challenge. We investigate how data diversification strategies in preference optimization can improve the mathematical reasoning abilities of large language models (LLMs). We evaluate three common data generation methods: temperature sampling, Chain-of-Thought prompting, and Monte Carlo Tree Search (MCTS), and introduce Diversified-ThinkSolve (DTS), a novel structured approach that systematically decomposes problems into diverse reasoning paths. Our results show that with strategically diversified preference data, models can substantially improve mathematical reasoning performance, with the best approach yielding gains of 7.1% on GSM8K and 4.2% on MATH over the base model. Despite its strong performance, DTS incurs only a marginal computational overhead (1.03x) compared to the baseline, while MCTS is nearly five times more costly with lower returns. These findings demonstrate that structured exploration of diverse problem-solving methods creates more effective preference data for mathematical alignment than traditional approaches.
BibTeX
@misc{dokmeci2025data,
  author = {Berkan Dokmeci and Qingyang Wu and Ben Athiwaratkun and Ce Zhang and Shuaiwen Leon Song and James Zou},
  title = {{Data Diversification Methods In Alignment Enhance Math Performance In LLMs}},
  year = {2025},
  eprint = {2507.02173},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.02173},
}
Figure