When a student solves a complex calculus problem, a teacher doesn't just look at the final number at the bottom of the page. They grade the derivation—the "process"—to ensure the student didn't arrive at the correct answer through a series of lucky mistakes.
In the world of Large Language Models (LLMs), we have traditionally relied on Outcome-Based Reward Models (ORM), which provide feedback only on the final result. However, a growing body of research suggests that for complex reasoning, we must shift toward Process-Based Reward Models (PRM). By rewarding individual reasoning steps, we can solve the "credit assignment problem," mitigate reward hacking, and unlock new scaling laws for "slow thinking" models.
The Credit Assignment Problem
The fundamental limitation of outcome-based supervision is the sparsity of the signal. As Pranjal Aggarwal and colleagues argue in RLHF Deciphered, sparse feedback makes it difficult for a model to know which specific action in a long sequence led to success or failure.
If a model generates a 50-line coding solution that fails a single test case, an ORM provides a binary "zero" reward. The model doesn't know if the logic was flawed from line one or if it simply missed a semicolon on line 49. Process supervision addresses this by evaluating every intermediate step. Hunter Lightman and the OpenAI team demonstrated in Let's Verify Step by Step that this granular signal allows models to pinpoint exactly where a logical error occurs, leading to a state-of-the-art 78.2% success rate on the challenging MATH benchmark.
Mitigating Reward Hacking
A dangerous phenomenon in LLM training is "right for the wrong reasons." Because models are optimized to maximize reward, they often find "shortcuts"—hallucinated or nonsensical logical steps that happen to terminate in the correct final answer. This is known as reward hacking.
Qi Pan notes in A Survey of Slow Thinking-based Reasoning LLMs that outcome-only evaluation encourages these shortcuts. By contrast, process-oriented rewards ensure that the model achieves the correct answer through a valid logical path. In specialized domains like medicine, this is critical. Linjie Mu introduced the Critical Evidence Graph (CEG) in MedCEG to ensure that medical AI doesn't just guess a diagnosis but provides a verifiable, clinically sound reasoning chain.
From Training Signal to Inference Navigator
Process rewards are not just for training; they are increasingly used at "test-time" to guide the model's thinking. This is the core of the "System 2" or "slow thinking" paradigm.
- Search and Pruning: In BPP-Search, Xiongwei Han uses a PRM as a "critic" to score partial solutions. If a reasoning path starts to look illogical, the search algorithm (like Beam Search or MCTS) can prune that branch early, saving compute for more promising paths.
- Dynamic Backtracking: Qianli Ma describes PRMs as "navigators" in Let's reward step by step. During inference, if the reward model detects a low-quality step, the system can backtrack and regenerate that specific step rather than restarting the entire problem.
- Ensembling: Instead of just averaging the final answers of different models, Sungjin Park introduced LE-MCTS, which uses a PRM to surgically pick the best individual reasoning steps from a pool of different models to construct a "super-solution."
The Scaling Law of Verification
Why is this shift happening now? It may be a mathematical necessity. Amrith Setlur proves in Scaling Test-Time Compute Without Verification or RL is Suboptimal that as we increase the computational budget at inference time, the gap between "verifier-based" methods (using PRMs) and "verifier-free" methods grows significantly.
Furthermore, process supervision helps models generalize from easy to hard problems. Zhiqing Sun found in Easy-to-Hard Generalization that reward models trained on the logic of simple problems can effectively supervise a generator to solve much more complex, competition-level challenges.
Challenges: The Cost of Granularity
Despite its benefits, process supervision is not a silver bullet. Sarah Pan observed a surprising trade-off in Let's Reinforce Step by Step: while PRMs boost performance on simpler math, they can sometimes hinder performance on highly complex problems compared to traditional outcome rewards. This may be due to the "narrower generalization range" of step-wise rewards, as noted in Does RLHF Scale?.
Additionally, the human cost is high. Training a PRM requires massive datasets of human-labeled steps—like the PRM800K dataset. To circumvent this, researchers are exploring automated verifiers like TinyV, a lightweight model designed specifically to catch false negatives in the reasoning process.
By shifting the focus from the "what" (the answer) to the "how" (the reasoning), process reward models are transforming LLMs from intuitive guessers into deliberate, verifiable problem solvers.