Abstract
Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the fine-grained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks.
Introduction
Fine-tuning Large Language Models (LLMs) for complex reasoning tasks typically relies on Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO). While effective for general dialogue, these methods often treat an entire multi-step solution as a single unit of feedback. In mathematical reasoning, this "coarse" approach is suboptimal; a single error in an intermediate step renders the entire solution incorrect, yet the model receives no granular signal on where the logic failed. Existing step-level datasets are prohibitively expensive to produce, often requiring manual annotation by experts or high-cost proprietary models.
To bridge this gap, we propose Step-level Value Preference Optimization (SVPO). SVPO shifts the focus from solution-level "all-or-nothing" feedback to granular, step-by-step evaluation, enabling models to identify specific points of failure and learn from self-generated exploration.
Methodology
The SVPO framework is built upon three core components designed to automate and refine the learning process for multi-step reasoning.
1. Autonomous Annotation via MCTS
Rather than relying on external labels, SVPO utilizes Monte Carlo Tree Search (MCTS) to explore various reasoning paths. When a path reaches the ground-truth answer, the success signal is backpropagated to assign $Q$-values to intermediate steps. This process allows the model to distinguish between correct steps and "dead ends" autonomously. Notably, the method constructs preference pairs not just from immediate siblings in the search tree, but also from "cousin" nodes at the same depth, providing a more robust signal for what constitutes a high-quality state.
2. The SVPO Loss Function
The training objective integrates three distinct signals:
- Step-level DPO: Aligns the policy by comparing pairs of "better" vs. "worse" reasoning steps.
- Value Margin Loss: Trains an explicit value model to ensure a significant score gap between correct and incorrect steps.
- Regularization: Synchronizes the explicit value model with the implicit reward of the DPO policy, preventing model degeneration during fine-tuning.
3. Step-level Beam Search (SBS)
During inference, the trained value model acts as a navigator. Instead of simple greedy decoding, the system employs Step-level Beam Search (SBS), evaluating multiple potential next steps and selecting the path with the highest predicted value. This transforms the LLM into a goal-oriented solver capable of pruning erroneous reasoning branches in real-time.
Experimental Results
We evaluated SVPO on standard benchmarks, including GSM8K and MATH, as well as Out-of-Domain (OOD) challenges like GaoKao2023 and OCWCourses.
State-of-the-Art Performance
As shown in the results table, SVPO (7B) achieves remarkable performance, often surpassing much larger or proprietary models. On the challenging MATH benchmark, SVPO reached 67.2% accuracy, significantly outperforming GPT-4's 42.5%.
Generalization and Search Gains
The model's robustness is particularly evident in OOD scenarios. On the undergraduate-level OCWCourses dataset, SVPO achieved 40.8%, a substantial lead over GPT-4 PAL (30.1%) and DeepSeekMath-Instruct (18.0%). Furthermore, the integration of Step-level Beam Search (SBS) consistently provided a 5-6% boost over greedy decoding, validating the effectiveness of the learned value model in guiding the reasoning process.
Conclusion
SVPO demonstrates that self-exploration through MCTS can effectively replace expensive human or proprietary annotations for mathematical reasoning. By optimizing at the step level and utilizing an explicit value model for search-based inference, SVPO enables 7B-parameter models to achieve state-of-the-art results. Future work will explore extending this framework to multimodal reasoning tasks involving visual and tabular data.
Summary updated