Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, Zhifang Sui

Abstract

Large Language Models (LLMs) have shown promise in mathematical reasoning but often struggle with logical consistency across multi-step solutions. Process Reward Models (PRMs) address this by providing step-by-step feedback, yet they typically rely on costly human annotations. We introduce Math-Shepherd, a process-oriented reward model trained on automatically constructed supervision data. By utilizing a "completer" model to estimate the potential of intermediate steps, Math-Shepherd enables effective verification and reinforcement learning without human intervention. Our approach significantly improves the performance of open-source LLMs, with DeepSeek-67B reaching 93.3% on GSM8K and 48.1% on MATH, rivaling or exceeding proprietary models like GPT-4.

Introduction

While LLMs excel at many tasks, mathematical reasoning remains a challenge due to the "black box" nature of their outputs. A model may produce a correct final answer through flawed logic or fail a complex problem due to a single early error. Traditional Outcome Reward Models (ORMs) only evaluate the final result, providing a "noisy" signal that fails to pinpoint where reasoning goes astray.

Process Reward Models (PRMs) offer a solution by grading every step of a reasoning chain. However, the primary bottleneck for PRMs has been the requirement for massive human-labeled datasets, such as OpenAI’s PRM800K. Math-Shepherd overcomes this by automating the generation of process-wise labels, allowing models to learn from their own reasoning paths.

Methodology: Automatic Process Annotation

The core of Math-Shepherd is an automated framework that assigns quality scores to intermediate reasoning steps based on their "potential" to lead to a correct solution.

The Math-Shepherd framework: (a) standard outcome annotation vs. (b) the proposed process annotation using multiple completions to estimate step quality.

As illustrated in the figure above, the process involves:

  1. Step-by-Step Generation: A math problem is solved by an LLM, producing a sequence of steps $s_1, s_2, \dots, s_k$.
  2. Path Completion: For each intermediate step $s_i$, a "completer" model generates $N$ independent paths to a final answer.
  3. Quality Estimation: The quality of step $s_i$ is determined by the success rate of these completions.
  4. Hard Estimation (HE): Assigns a binary score (1 if any completion is correct, 0 otherwise).
  5. Soft Estimation (SE): Assigns a continuous score based on the ratio of correct completions to total completions.

The resulting dataset is used to train the Math-Shepherd reward model, which can then be applied for Verification (reranking multiple model outputs) or Reinforcement Learning (using Proximal Policy Optimization to reward correct reasoning steps during training).

Experimental Results

We evaluated Math-Shepherd across several benchmarks, including GSM8K (grade school math) and MATH (competition-level problems). The integration of Math-Shepherd led to substantial performance gains across various model architectures.

Performance of various fine-tuned LLMs on the GSM8K benchmark with and without the Math-Shepherd verification (+SHEPHERD).

As shown in the GSM8K results, models like DeepSeek-67B and LLaMA2-70B see significant improvements when paired with Math-Shepherd. Specifically, DeepSeek-67B's accuracy rose from 82.8% to 93.3%, surpassing the early GPT-4 baseline. Similar trends were observed on the more challenging MATH benchmark, where DeepSeek-67B improved from 36.8% to 48.1%.

Furthermore, reinforcement learning with step-by-step PPO proved highly effective. Mistral-7B's accuracy jumped from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH after PPO training with Math-Shepherd.

Discussion and Insights

A key finding of this work is that automated labels can be more effective than human labels due to distributional alignment. Because the labels are generated by models similar to those being trained, the reward signal is better suited to the model's internal logic. Additionally, PRMs demonstrate superior data efficiency; the dense feedback provided by step-by-step rewards allows models to learn more from fewer examples compared to outcome-only feedback.

Despite these successes, the approach has limitations. The "completion" process is computationally intensive, requiring multiple generations for every step in the training set. There is also inherent noise in automated labels if the completer model is not sufficiently capable. Future work will explore more efficient inference techniques and the potential of hybrid human-AI annotation systems.

Conclusion

Math-Shepherd demonstrates that high-quality process supervision can be achieved without the high cost of human annotation. By shifting the focus from final outcomes to the utility of intermediate reasoning steps, we can significantly enhance the mathematical capabilities and reliability of open-source LLMs.

Summary updated

Abstract

In this paper, we present an innovative process-oriented math process reward model called \textbf{Math-Shepherd}, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-Shepherd in two scenarios: 1) \textit{Verification}: Math-Shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) \textit{Reinforcement Learning}: Math-Shepherd is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With Math-Shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with Math-Shepherd significantly improves the accuracy of Mistral-7B (77.9\%$\to$84.1\% on GSM8K and 28.6\%$\to$33.0\% on MATH). The accuracy can be further enhanced to 89.1\% and 43.5\% on GSM8K and MATH with the verification of Math-Shepherd, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
BibTeX
@inproceedings{wang2024math,
  author = {Peiyi Wang and Lei Li and Zhihong Shao and R. X. Xu and Damai Dai and Yifei Li and Deli Chen and Y. Wu and Zhifang Sui},
  title = {{Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=o5P5WVctK0},
}
Figure