Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision

Tej Deep Pala, Pritha Sharma, Amir Zadeh, Chuan Li, Soujanya Poria2025 · DOI

Abstract

Large Language Models (LLMs) are prone to hallucination, especially during multi-hop and reasoning-intensive tasks such as mathematical problem solving. While Outcome Reward Models verify only final answers, Process Reward Models (PRMs) score each intermediate step to steer generation toward coherent solutions. We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM that first classifies math and consistency errors at each step, then combines these fine-grained signals to estimate step correctness. To train PathFinder-PRM, we construct a 400K-sample dataset by enriching the human-annotated PRM800K corpus and RLHFlow Mistral traces with three-dimensional step-level labels. On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3x less data. When applied to reward-guided greedy search, our model yields prm@8 48.3, a +1.5 point gain over the strongest baseline. These results demonstrate that decoupled error detection and reward estimation not only boost fine-grained error detection but also substantially improve end-to-end, reward-guided mathematical reasoning with greater data efficiency.

The Challenge: The PRM "Blind Spot"

Large Language Models frequently "hallucinate" or make subtle logical slips during multi-step reasoning. Traditional Outcome Reward Models (ORMs) only check if the final answer is correct, which is often too late to fix a straying thought process. Process Reward Models (PRMs) were designed to solve this by scoring every individual step.

However, current PRMs struggle to distinguish between two distinct signals: factual/logical errors and path optimality. Existing models often fail to catch subtle issues like redundant steps or domain inconsistencies because they attempt to predict a single "correctness" score for complex reasoning steps.

Methods: Decoupling Error Detection

PathFinder-PRM introduces a hierarchical, two-pass inference strategy that treats error detection and value estimation as complementary but distinct tasks:

  1. Pass 1 (Error Typing): The model performs a forward pass to predict the probability of two specific error types: Math Errors (arithmetic slips, wrong formulas) and Consistency Errors (logical contradictions with the question or previous steps).
  2. Pass 2 (Reward Estimation): These predicted error labels are re-inserted into the input. The model then performs a second forward pass to estimate the final Step Reward, explicitly conditioned on the detected errors.
Comparison of PathFinder-PRM against existing PRMs, showing how it identifies specific error types (Math vs. Consistency) to provide more accurate rewards.

Architecturally, the model is initialized from Qwen2.5-Math-7B-Instruct and uses special tokens (<+> and <->) for classification. This decoupled design prevents error cascading and provides more interpretable supervision.

Data and Training

The researchers curated a 400K-sample dataset with three-dimensional step-level labels, utilizing the human-annotated PRM800K corpus (345K trajectories) and RLHFlow Mistral traces (55K trajectories). To generate fine-grained labels for erroneous steps, they employed DeepSeek-R1-Distill-Qwen-32B as a judge, ensuring quality through consistency filtering against existing Monte Carlo estimates.

Key Results

PathFinder-PRM achieves state-of-the-art performance across multiple benchmarks:

  • SOTA on PRMBench: Achieved a PRMScore of 67.7, surpassing the previous best of 65.5.
  • ProcessBench Excellence: On the PRM800K-only subset, it outperformed the previous best by 6.7 points (65.2 vs 58.5).
  • End-to-End Reasoning: When guiding a greedy search, it achieved a prm@8 score of 48.3, a +1.5 point gain over the strongest baseline.
PathFinder-PRM variants demonstrate superior data efficiency, achieving higher overall PRM scores with significantly less training data than baseline models.

A standout finding is the model's data efficiency. PathFinder-PRM outperformed models trained on 1.5 million samples while using only 400K samples. This suggests that the granularity of supervision—specifically distinguishing between calculation errors and consistency errors—is more critical than raw data volume.

Conclusion

By decoupling error detection from reward estimation, PathFinder-PRM provides a more robust and interpretable signal for mathematical reasoning. The model shows significant improvements in detecting "subtle" errors like redundancy and circular logic where traditional PRMs typically fail. Future work may explore expanding error categories and integrating this hierarchical feedback directly into Reinforcement Learning pipelines.

Summary updated

Abstract

Large Language Models (LLMs) are prone to hallucination, especially during multi-hop and reasoning-intensive tasks such as mathematical problem solving.While Outcome Reward Models verify only final answers, Process Reward Models (PRMs) score each intermediate step to steer generation toward coherent solutions.We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM that first classifies math and consistency errors at each step, then combines these fine-grained signals to estimate step correctness.To train PathFinder-PRM, we construct a 400K-sample dataset by enriching the human-annotated PRM800K corpus and RLHFlow Mistral traces with three-dimensional step-level labels.On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 less data.When applied to reward guided greedy search, our model yields prm@8 48.3, a +1.5 point gain over the strongest baseline.These results demonstrate that decoupled error detection and reward estimation not only boost fine-grained error detection but also substantially improve end-to-end, reward-guided mathematical reasoning with greater data efficiency. 1 .
BibTeX
@misc{pala2025error,
  author = {Tej Deep Pala and Pritha Sharma and Amir Zadeh and Chuan Li and Soujanya Poria},
  title = {{Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.807},
}
Figure