Abstract
In this paper, we present an innovative process-oriented math process reward model called Math-shepherd, which assigns a reward score to each step of math problem solutions. The training of Math-shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-shepherd in two scenarios: 1) Verification: Math-shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) Reinforcement Learning (RL): Math-shepherd is employed to reinforce LLMs. With Math-shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, process RL with Math-shepherd significantly enhances Mistral-7B (77.9% → 84.1% on GSM8K and 28.6% → 33.0% on MATH). The accuracy can be further improved to 89.1% and 43.5% on two benchmarks with verification of Math-shepherd. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
Introduction
While Large Language Models (LLMs) have become adept at many tasks, multi-step mathematical reasoning remains a significant hurdle. Errors often "hallucinate" early in a derivation, leading to an incorrect final answer. Standard Outcome Reward Models (ORMs) only grade the final answer, providing a coarse signal that fails to pinpoint where reasoning went astray. Process Reward Models (PRMs) are more effective as they provide feedback on every individual step; however, existing PRMs (like OpenAI’s PRM800K) rely on massive amounts of expensive, expert-level human annotations. Math-Shepherd breaks this bottleneck by automating process-level supervision, enabling the training of high-quality PRMs without human intervention.
Methodology
The core innovation of Math-Shepherd is an Automatic Process Annotation framework inspired by Monte Carlo Tree Search (MCTS). The process involves two primary stages:
- Data Construction: For a given math problem and a partial solution up to step $i$, a fine-tuned LLM (the "completer") generates $N$ different completions to reach a final answer.
- Potential Estimation: The quality of step $i$ is defined by its "potential" to reach the correct answer. The authors utilize two methods: Hard Estimation (HE), a binary label indicating if any completion is correct, and Soft Estimation (SE), a continuous score based on the frequency of correct completions.
This automatically labeled data is used to train the Math-Shepherd PRM, which is then applied for Verification (reranking candidate solutions at inference) and Reinforcement Learning (using PPO to provide step-wise rewards during training).
Experimental Results
Math-Shepherd was evaluated on standard benchmarks including GSM8K and MATH, as well as out-of-distribution (OOD) tests like the Hungarian National Final Exam. The results demonstrate that process-level supervision is significantly more data-efficient and effective than outcome-level supervision.
Key findings include:
- Reinforcement Learning Gains: Applying Process RL to Mistral-7B improved its performance on GSM8K from 77.9% to 84.1% and on MATH from 28.6% to 33.0%.
- Verification Power: When used to rerank 256 samples, the RL-tuned Mistral-7B reached 89.1% on GSM8K and 43.5% on MATH.
- State-of-the-Art for Open Source: Math-Shepherd as a verifier for DeepSeek-67B achieved 93.3% on GSM8K and 48.1% on MATH, setting new benchmarks for open-source models not using external tools.
- Distribution Alignment: The automated PRM often outperformed human-annotated models because it could be trained on the exact model family it was intended to verify, closing the "distribution gap" inherent in cross-model human labeling.
Conclusion
Math-Shepherd demonstrates that automatic process supervision is a viable and powerful alternative to human annotation for mathematical reasoning. By providing dense, step-wise feedback, it enables open-source models to reach performance levels previously reserved for closed-source systems. Future work may explore iterative training cycles where the reward model and policy are updated concurrently to further refine reasoning capabilities.
Summary updated