The Bidirectional Process Reward Model

Lingyin Zhang, Jun Gao, Xiaoxue Ren, Ziqiang Cao

Abstract

Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promising approach to enhance the reasoning quality of Large Language Models (LLMs). However, most existing PRMs rely on a unidirectional left-to-right (L2R) evaluation scheme, which restricts their utilization of global context. In light of this challenge, we propose a novel bidirectional evaluation paradigm, named Bidirectional Process Reward Model (BiPRM). BiPRM incorporates a parallel right-to-left (R2L) evaluation stream, implemented via prompt reversal, alongside the conventional L2R flow. Then a gating mechanism is introduced to adaptively fuse the reward scores from both streams to yield a holistic quality assessment. Remarkably, compared to the original PRM, BiPRM introduces only a 0.3% parameter increase for the gating module, and the parallel execution of two streams incurs merely 5% inference time latency. Our extensive empirical evaluations spanning diverse benchmarks, LLM backbones, PRM objectives and sampling policies demonstrate that BiPRM consistently surpasses unidirectional baselines, achieving an average relative gain of 10.6% over 54 solution-level configurations and 37.7% in 12 step-level error detection scenarios. Generally, our results highlight the effectiveness, robustness and general applicability of BiPRM, offering a promising new direction for process-based reward modeling.

Introduction: The Challenge of Unidirectional Evaluation

As Large Language Models (LLMs) are increasingly deployed for complex reasoning tasks in mathematics and coding, the need for reliable verification has grown. Process Reward Models (PRMs) have emerged as a solution, providing step-by-step supervision rather than just scoring a final answer. However, traditional PRMs are almost exclusively unidirectional, processing information from left-to-right (L2R).

This unidirectional approach suffers from a lack of global context. In many reasoning chains, the validity of an early step—such as a conjecture or a variable assignment—can only be fully verified by observing its downstream consequences. A standard PRM evaluating Step 2 only sees the Question and Step 1, effectively operating with a "blind spot" regarding the future of the trajectory. This limitation often leads to false positives where logically flawed early steps are given high scores because they appear superficially coherent.

The BiPRM Framework

To address this, we introduce the Bidirectional Process Reward Model (BiPRM). BiPRM draws inspiration from classical bidirectional architectures but adapts them for the transformer era through an elegant dual-stream approach.

Comparison of unidirectional scoring (limited context) versus the proposed holistic bidirectional scoring.

BiPRM operates two parallel evaluation flows:

  1. L2R Stream: The conventional forward flow evaluating steps based on preceding context.
  2. R2L Stream: A retrospective flow implemented via prompt reversal. By flipping the order of reasoning steps in the prompt, the model is forced to look from the conclusion back toward the premises, treating early steps as terminal steps with full visibility of the result.

These streams are unified by a Dynamic Gating Mechanism. A lightweight Multi-Layer Perceptron (MLP) computes a dynamic weight for each step, allowing the model to adaptively fuse scores based on which stream is more trustworthy at a given point in the reasoning chain. Despite the dual-stream nature, parallel execution on modern GPUs keeps inference latency to a negligible 5% increase.

Analyzing the Power of Hindsight

Our analysis reveals that L2R and R2L streams possess highly complementary error profiles. L2R models naturally become more accurate toward the end of a solution as context accumulates. Conversely, R2L models are significantly more effective at catching errors in the early stages of a solution because they view those steps through the lens of the final outcome.

Average MAE and gating weights across reasoning progress. BiPRM (red) consistently achieves lower error than unidirectional baselines by leveraging complementary stream strengths.

As shown in the figure above, the gating mechanism learns to shift its reliance as reasoning progresses. The R2L stream provides a critical "hindsight" signal that prevents the model from being deceived by early logical fallacies that might otherwise seem plausible in a forward-only pass.

Empirical Performance

BiPRM was evaluated across several rigorous benchmarks, including GSM-Plus, MATH500, and ProcessBench, using backbones ranging from 1B to 7B parameters. The results demonstrate "universal improvements":

  • Solution Accuracy: BiPRM achieved an average relative gain of 10.6% in solution-level accuracy (Best-of-N) across 54 different experimental configurations.
  • Error Localization: On the diagnostic ProcessBench task, BiPRM outperformed unidirectional baselines by 37.7% in F1 score. This highlights its superior ability to pinpoint exactly where a reasoning chain first deviates from logic.
  • Robustness: The model showed consistent gains regardless of the underlying training objective (BCE, MSE, or Ranking loss), proving the architectural benefit of bidirectionality is independent of the specific loss function used.

Conclusion and Future Work

BiPRM offers a robust and efficient path forward for process-based reward modeling. By giving LLMs the power of hindsight through prompt reversal and dynamic gating, we significantly improve their ability to verify complex reasoning. While currently focused on mathematical domains, future work will explore extending this bidirectional paradigm to open-ended common sense reasoning and symbolic logic, as well as addressing "internally consistent" errors where a model's logic remains sound even after an initial incorrect premise.

Summary updated

Abstract

Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promising approach to enhance the reasoning quality of Large Language Models (LLMs). However, most existing PRMs rely on a unidirectional left-to-right (L2R) evaluation scheme, which restricts their utilization of global context. In light of this challenge, we propose a novel bidirectional evaluation paradigm, named Bidirectional Process Reward Model (BiPRM). BiPRM incorporates a parallel right-to-left (R2L) evaluation stream, implemented via prompt reversal, alongside the conventional L2R flow. Then a gating mechanism is introduced to adaptively fuse the reward scores from both streams to yield a holistic quality assessment. Remarkably, compared to the original PRM, BiPRM introduces only a 0.3% parameter increase for the gating module, and the parallel execution of two streams incurs merely 5% inference time latency. Our extensive empirical evaluations spanning diverse benchmarks, LLM backbones, PRM objectives and sampling policies demonstrate that BiPRM consistently surpasses unidirectional baselines, achieving an average relative gain of 10.6% over 54 solution-level configurations and 37.7% in 12 step-level error detection scenarios. Generally, our results highlight the effectiveness, robustness and general applicability of BiPRM, offering a promising new direction for process-based reward modeling.
BibTeX
@inproceedings{zhang2026bidirectional,
  author = {Lingyin Zhang and Jun Gao and Xiaoxue Ren and Ziqiang Cao},
  title = {{The Bidirectional Process Reward Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.572},
}
Figure