Abstract
Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the fine-grained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks.
Introduction
While Large Language Models (LLMs) have achieved remarkable success across various domains, complex multi-step reasoning—such as solving high-school competition-level mathematics—remains a significant challenge. Current alignment techniques like Direct Preference Optimization (DPO) typically rely on solution-level preferences. This "all-or-nothing" feedback approach informs the model whether a complete answer is correct but fails to pinpoint exactly where a mistake occurred in a multi-step derivation.
This creates a "credit assignment" problem: a model might penalize a logically sound sequence because of a minor final calculation error, or conversely, reward a flawed reasoning path that happened to arrive at the correct answer. To bridge this gap, we propose Step-level Value Preference Optimization (SVPO), a framework designed to provide fine-grained, step-level feedback without the prohibitive cost of human expert labeling.
The SVPO Framework
SVPO moves beyond solution-level feedback by integrating automated annotation with a multi-objective training strategy.
Automated Step-level Annotation
Instead of relying on external proprietary models for annotation, SVPO utilizes Monte Carlo Tree Search (MCTS) for self-exploration. The model explores various reasoning paths for a given problem; steps that consistently lead to correct final answers are assigned higher Q-values. By comparing sibling nodes within the search tree, the system automatically identifies "preferred" versus "rejected" step pairs, creating a rich dataset for step-level learning.
Multi-Objective Optimization
The SVPO training objective consists of three primary components:
- Step-level DPO: Applies preference optimization logic to individual reasoning steps.
- Value Margin Loss: Trains an explicit value head to rank "good" steps higher than "bad" steps by a defined margin ($\gamma$).
- Regularization MSE: A novel term ensuring the explicit value model remains aligned with the implicit reward defined by the policy's log-likelihoods, preventing model degeneration.
During inference, we employ Step-level Beam Search (SBS). The trained value model acts as a navigator, evaluating multiple candidate steps and steering the generation toward the most promising logical paths.
Experimental Evaluation
We evaluated SVPO on 7B-parameter models across several rigorous mathematical benchmarks, including in-domain sets (GSM8K, MATH) and out-of-domain (OOD) sets (GaoKao2023, OCWCourses). The training data comprised 15k questions, which MCTS expanded into 56k positive reasoning instances.
Key Results
As shown in the performance comparison below, SVPO achieves state-of-the-art results for 7B-parameter models:
- State-of-the-Art Performance: On the MATH benchmark, SVPO reached 67.2% accuracy, significantly outperforming the previous best open-source model, DeepSeekMath-Instruct (57.4%).
- Surpassing Proprietary Models: On the challenging OCWCourses benchmark (MIT OpenCourseWare), SVPO achieved 40.8%, notably higher than GPT-4's 30.1% using Program-Aided Language (PAL).
- OOD Robustness: The model demonstrated strong generalization on the GaoKao2023 dataset (55.3%), suggesting that step-level learning fosters genuine logical reasoning rather than pattern memorization.
Discussion and Conclusion
Our results highlight two critical insights. First, self-exploration via MCTS is superior to simple distillation from larger models because it targets the specific errors the current model is prone to making. Second, the explicit value model serves as an effective navigator during inference, pruning dead-end reasoning paths before they are fully explored.
Ablation studies confirmed that removing the Margin Loss or MSE Regularization leads to significant performance degradation, validating the importance of our multi-objective approach. While SVPO is more computationally intensive than standard supervised fine-tuning, the gains in reasoning accuracy and robustness are substantial. Future work will explore extending SVPO to multimodal reasoning tasks involving geometric or tabular data.
Summary updated