Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, Zhifang Sui

Abstract

In this paper, we present an innovative process-oriented math process reward model called Math-shepherd, which assigns a reward score to each step of math problem solutions. The training of Math-shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-shepherd in two scenarios: 1) Verification: Math-shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) Reinforcement Learning (RL): Math-shepherd is employed to reinforce LLMs. With Math-shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, process RL with Math-shepherd significantly enhances Mistral-7B (77.9% → 84.1% on GSM8K and 28.6% → 33.0% on MATH). The accuracy can be further improved to 89.1% and 43.5% on two benchmarks with verification of Math-shepherd. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.

Introduction

While Large Language Models (LLMs) have become adept at many tasks, multi-step mathematical reasoning remains a significant hurdle. Errors often "hallucinate" early in a derivation, leading to an incorrect final answer. Standard Outcome Reward Models (ORMs) only grade the final answer, providing a coarse signal that fails to pinpoint where reasoning went astray. Process Reward Models (PRMs) are more effective as they provide feedback on every individual step; however, existing PRMs (like OpenAI’s PRM800K) rely on massive amounts of expensive, expert-level human annotations. Math-Shepherd breaks this bottleneck by automating process-level supervision, enabling the training of high-quality PRMs without human intervention.

Methodology

The core innovation of Math-Shepherd is an Automatic Process Annotation framework inspired by Monte Carlo Tree Search (MCTS). The process involves two primary stages:

  1. Data Construction: For a given math problem and a partial solution up to step $i$, a fine-tuned LLM (the "completer") generates $N$ different completions to reach a final answer.
  2. Potential Estimation: The quality of step $i$ is defined by its "potential" to reach the correct answer. The authors utilize two methods: Hard Estimation (HE), a binary label indicating if any completion is correct, and Soft Estimation (SE), a continuous score based on the frequency of correct completions.
The Automatic Process Annotation framework, showing how partial solutions are completed and scored using Hard and Soft Estimation.

This automatically labeled data is used to train the Math-Shepherd PRM, which is then applied for Verification (reranking candidate solutions at inference) and Reinforcement Learning (using PPO to provide step-wise rewards during training).

Experimental Results

Math-Shepherd was evaluated on standard benchmarks including GSM8K and MATH, as well as out-of-distribution (OOD) tests like the Hungarian National Final Exam. The results demonstrate that process-level supervision is significantly more data-efficient and effective than outcome-level supervision.

Performance of Math-Shepherd (SHEPHERD) compared to Outcome Reward Models (ORM) and Self-Consistency (SC) across GSM8K and MATH500 benchmarks.

Key findings include:

  • Reinforcement Learning Gains: Applying Process RL to Mistral-7B improved its performance on GSM8K from 77.9% to 84.1% and on MATH from 28.6% to 33.0%.
  • Verification Power: When used to rerank 256 samples, the RL-tuned Mistral-7B reached 89.1% on GSM8K and 43.5% on MATH.
  • State-of-the-Art for Open Source: Math-Shepherd as a verifier for DeepSeek-67B achieved 93.3% on GSM8K and 48.1% on MATH, setting new benchmarks for open-source models not using external tools.
  • Distribution Alignment: The automated PRM often outperformed human-annotated models because it could be trained on the exact model family it was intended to verify, closing the "distribution gap" inherent in cross-model human labeling.

Conclusion

Math-Shepherd demonstrates that automatic process supervision is a viable and powerful alternative to human annotation for mathematical reasoning. By providing dense, step-wise feedback, it enables open-source models to reach performance levels previously reserved for closed-source systems. Future work may explore iterative training cycles where the reward model and policy are updated concurrently to further refine reasoning capabilities.

Summary updated

Abstract

In this paper, we present an innovative process-oriented math process reward model called Math-shepherd, which assigns a reward score to each step of math problem solutions. The training of Math-shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-shepherd in two scenarios: 1) Verification: Math-shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) Reinforcement Learning (RL): Math-shepherd is employed to reinforce LLMs.With Math-shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, process RL with Math-shepherd significantly enhances Mistral-7B (77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH).The accuracy can be further improved to 89.1% and 43.5% on two benchmarks with verification of Math-shepherd.We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
BibTeX
@inproceedings{wang2024math,
  author = {Peiyi Wang and Lei Li and Zhihong Shao and Runxin Xu and Damai Dai and Yifei Li and Deli Chen and Yu Wu and Zhifang Sui},
  title = {{Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=o5P5WVctK0},
}
Figure