GPRM: Global Perspective Process Reward Model

Zifan Zhang, Zhengbo Jiao, Shaobo Wang, Jing Zhou, Guoyi Li, Jiayi Chen, Wei Wang, Bing Zhao, Cheng Fang, Xiaoxiao Xu, Hu Wei, Linfeng Zhang, Lin QuarXiv 2026 · arXiv

TL;DR

GPRM (Global Perspective Process Reward Model) revolutionizes step-level reasoning evaluation by moving beyond local context to a "global" view that incorporates historical judgments and future reasoning trajectories. By analyzing steps through four diagnostic dimensions—consistency, plausibility, validity, and alignment—GPRM outperforms proprietary giants like GPT-4o and o1-mini on reasoning benchmarks despite having a significantly smaller parameter count.

What Problem Does This Paper Address?

Large Language Models (LLMs) often fail in multi-step reasoning due to subtle intermediate errors that compound over time. To fix this, researchers use Process Reward Models (PRMs) to grade each individual step rather than just the final answer.

However, existing PRMs suffer from two major "blind spots":

  1. Lack of Historical Inheritance: They often evaluate the current step without explicitly remembering the judgments made on previous steps, leading to inconsistent "broken" reasoning chains.
  2. Lack of Global Awareness: They judge a step based only on what came before it. This penalizes "aha-moments"—exploratory detours or creative leaps that might look like errors locally but are actually necessary turning points that lead to the correct solution.

Methods

The authors introduce GPRM, which redefines the input for a reward model. Instead of just looking at the question and the current step, GPRM evaluates a step $s_i$ by conditioning on a "Global Perspective": $(Question, Historical Judgments, Current Step, Future Trajectory)$.

The Four-Dimensional Diagnostic Framework

GPRM doesn't just output a single score; it performs a structured analysis across four axes:

  • Look-back (Forward Consistency): Does this step contradict what we already established?
  • Look-ahead (Posterior Verification): Does this step actually prove useful in the subsequent reasoning?
  • Self-check (Self Validity): Is the math/logic in this specific step sound?
  • Goal Alignment: Is the model still trying to solve the original problem, or has it drifted off-topic?

Training Pipeline

The model follows a two-stage process:

  1. Supervised Fine-Tuning (SFT): The model is trained to imitate a "teacher" LLM that provides detailed reasoning for its step-level grades.
  2. Reinforcement Learning (GRPO): Using Group Relative Policy Optimization, the model is further refined using outcome-based rewards, focusing specifically on "hard" examples where the model previously struggled.

Datasets

  • Training: The authors used OpenMathInstruct-2 for the SFT phase (with targeted error injection) and PRM800K for the GRPO phase.
  • Evaluation: The model was tested on PRMBench and ProcessBench (for math reasoning), Agent Error Bench (for cross-task generalization in interactive environments), and PPE (Preference Proxy Evaluations).

Key Insights and Ideas

  • Hindsight is 20/20: The most striking insight is that a step's "correctness" is often only visible once you see where it leads. By completing the trajectory before judging the intermediate step, GPRM reduces mis-localization of errors (penalizing the wrong step) from 37.6% to 15.7%.
  • Judgment Inheritance: By feeding previous judgments back into the model (not just the previous text), the model maintains a coherent "memory" of the error chain, preventing it from hallucinating that a reasoning path has been "fixed" when it hasn't.
  • Generative vs. Discriminative: Moving from a simple classifier (Correct/Incorrect) to a generative model that explains its diagnosis significantly improves the robustness and interpretability of the reward signal.

Key Results

  • Benchmark Dominance: GPRM-4B achieved a score of 73.9 on PRMBench and 74.3 on ProcessBench, surpassing GPT-4o (66.8 / 61.9) and o1-mini (68.8 / 87.9 on ProcessBench).
  • Scaling Success: The 14B version of GPRM further pushed these leads, showing that the global perspective framework scales effectively with model size.
  • Agent Generalization: Despite being trained primarily on math, GPRM generalized to agent trajectories (like ALFWorld and GAIA), outperforming GPT-4.1-based baselines in identifying the earliest error step.
  • Test-Time Scaling: When used to guide a search (Greedy Guided Search@8), GPRM improved final answer accuracy by 5.6 points, proving its utility as a high-quality verifier.

Limitations and Future Work

  • Inference Overhead: The "Global Perspective" comes at a cost. To judge a step in an online setting (where the future isn't yet written), the model must first generate a potential future trajectory. This increases latency and token usage (up to +71.7% tokens in greedy search).
  • Teacher Dependency: The SFT phase relies heavily on the quality of the teacher model's explanations. If the teacher provides flawed reasoning for a correct label, the PRM may inherit those biases.
  • Future Directions: The authors suggest exploring more efficient ways to gain a "global view" without full trajectory completion and extending the framework to even more diverse, non-mathematical reasoning tasks.

Summary updated

Abstract

Process Reward Models (PRMs) are widely used for step-level evaluation of multi-step reasoning in large language models, but existing methods are often limited by weak historical dependency modeling and insufficient global awareness. They typically fail to leverage prior step evaluations for coherent judgment and lack visibility into subsequent reasoning, leading to suboptimal assessment of intermediate steps. To address this issue, we propose the Global Perspective Process Reward Model (GPRM), which jointly models the problem, previous reasoning steps and their evaluations, the current step, and future steps to enable history-aware and globally informed process evaluation. We further introduce a structured evaluation framework along four dimensions---look back, look ahead, self-check, and goal alignment---to assess prior consistency, future plausibility, step validity, and task alignment. Based on this framework, we construct datasets for supervised fine-tuning (SFT) and reinforcement learning (GRPO). Experiments show that, despite using less training data and smaller model sizes, GPRM outperforms strong baseline models by 7.8 and 5.4 points on PRMBench and ProcessBench, respectively. GPRM also generalizes well across tasks, surpassing the GPT-4.1 baseline on Agent Error Bench, improving strong baselines by around 4 points under test-time scaling (TTS), and achieving competitive performance on PPE.
BibTeX
@misc{zhang2026gprm,
  author = {Zifan Zhang and Zhengbo Jiao and Shaobo Wang and Jing Zhou and Guoyi Li and Jiayi Chen and Wei Wang and Bing Zhao and Cheng Fang and Xiaoxiao Xu and Hu Wei and Linfeng Zhang and Lin Qu},
  title = {{GPRM: Global Perspective Process Reward Model}},
  year = {2026},
  eprint = {2611.20755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20755},
}
Figure