Do Process Reward Models Check Logic or Position?

A process reward model (PRM) grades each step of a model's written reasoning rather than only the final answer. PRMs are increasingly used to pick the best of many sampled solutions and to steer search, and their verdicts are trusted as if they reflected logical validity. We propose to test that trust directly. We generate reasoning problems from random computation graphs, so the correctness of every intermediate step can be computed exactly and no problem has appeared in any training corpus. On these problems we vary three things independently: the order in which premises are stated, the position in the chain where an error is planted, and how familiar the problem's surface wording looks. If a PRM's verdict on a step changes when only premise order or error position changes, then the PRM is partly scoring layout, not logic. That kind of systematic bias is the mechanism suspected of making PRM-guided search get worse as more samples are drawn.

The opportunity is that two problems usually studied apart, order sensitivity and benchmark contamination, both distort every existing PRM evaluation. Both can be removed at once with a procedural generator that also produces free, exact step labels. One error position gets special attention: the handoff from a free-form reasoning draft to the extracted final answer. Prior work shows reasoning models often answer against their own drafts there, and no one has checked whether PRMs notice. The deliverable is an order- and contamination-controlled audit of current PRMs and thinking critics. We also train a PRM on the generator's exact labels across many premise orderings, and we test whether it transfers to human-labeled static benchmarks.

Core Claim

We hypothesize that current PRMs detect step errors at rates that depend substantially on two kinds of position. The first is the position of the premises the step relies on, with a drop when they are stated in reverse dependency order. The second is the error's position in the chain, with errors at the draft-to-answer handoff the hardest to catch. We also expect detection to be inflated on surface forms resembling public math benchmarks. If the hypothesis holds, a PRM trained on generated traces with exact labels, seen under many premise orderings, should close most of these gaps. It should also improve error localization on ProcessBench, which it never trained on, and keep Best-of-N accuracy from degrading as the number of samples grows. The new element is not the generator, the PRM, or permutation augmentation taken alone. It is using exact, contamination-free step labels to separate a verifier's sensitivity to validity from its sensitivity to layout and familiarity, and then testing whether removing the latter fixes search behavior.

Grounding in Prior Work

Two results motivate the audit. DyVal generates reasoning problems from directed acyclic graphs, meaning dependency graphs with no cycles, and shows that reversing premise order costs GPT-4 13.6% accuracy on reachability. It also shows that models strong on static benchmarks score near zero on generated instances. DyVal checks only final answers and never scores intermediate steps. We reuse its generator as the source of exact step labels. Reward Under Attack audits Qwen2.5-Math-PRM-7B and Skywork-o1-Open-PRM-7B under eight perturbations and finds them nearly invariant to semantics-preserving edits (|ΔR| below 0.02) but blind to fluent hallucinations.

That invariance looks like counter-evidence, and we address it head on. The perturbations it tested were within-step reordering and question shuffling. Neither varies the order of premises a step depends on, and neither varies where in the chain an error sits. Score stability under rewording says nothing about whether detection survives reordering. ProcessBench supplies our transfer evaluation and our strongest baselines. It shows that 51.8% of Omni-MATH solutions with correct final answers contain errors, and that thinking critics beat scalar PRMs. It does not examine error position, and it is static.

The two-stage angle comes from Measuring the Faithfulness of Thinking Drafts. That paper shows reasoning models often extract answers that contradict their own drafts, with draft-answer consistency as low as 19.5% for QwQ on GPQA, but it only measures this by intervention and never asks whether a verifier would catch it. Work on decoupling free-form drafting from constrained extraction (Let Me Speak Freely?, Large Language Models are Zero-Shot Reasoners) makes the extraction step a distinct, auditable unit, so we treat it as a step position of its own.

Two papers supply the familiarity manipulation. ABCD uses meaningless content (NonsenseQA) to expose artifact exploitation and uses perplexity on original versus permuted orderings to reveal MMLU-Pro memorization. Its fixes target answer labels in multiple choice; we carry the idea over to verifiers. VersaPRM is our main comparison for synthetic step supervision: about 487,000 steps labeled by an LLM conditioned on the ground truth, drawn from MMLU-Pro. Its labels are approximate, and its source questions are of the kind ABCD shows to be contaminated. J1 removes order bias in pairwise judges with consistency rewards. Its judge emits a verdict on a whole response, not on individual steps, and it is evaluated on static benchmarks.

Research Program

The generator extends DyVal's arithmetic, Boolean-logic, and reachability graphs at depths D1–D4. Each graph node becomes one derivation step, and the final step restates the computed target value as an extracted answer. We produce correct traces by executing the graph. We produce flawed traces by planting a single error of one of three types. A value error gives a wrong node value. A dependency error uses a premise that does not feed the node. A handoff error leaves the draft correct but extracts an answer different from the draft's conclusion, mirroring the inconsistency the faithfulness study found. Because the graph is known, the earliest error index is exact.

For every trace we render the premises in topological, reversed, and random order, while keeping the step text fixed. We render the surface in three styles. One mimics public math benchmarks: word problems with everyday entities in the style of GSM8K. One is neutral symbolic. One uses invented nonsense nouns in the spirit of NonsenseQA. We additionally plant errors at controlled relative depths so that position and error type are crossed.

The audit fits a simple regression of detection outcome on error position, premise order, surface style, and graph depth for each verifier. It also reports flip rate: the fraction of steps whose correct/incorrect verdict changes across premise orderings of the same trace. For training, we fine-tune from the Qwen2.5-Math-PRM-7B checkpoint on generated traces, with each trace shown under several orderings and labeled exactly. Training uses generated graphs at sizes and seeds disjoint from evaluation. Finally, we run Best-of-N selection for N up to 256 over samples from an open reasoning model on held-out generated problems, using each verifier. We plot accuracy against N to see whether the degradation at large N, which Fei Yu et al. attribute to verifier bias, tracks each verifier's measured order sensitivity.

Baselines

The do-nothing reference is majority voting over final answers, together with the raw off-the-shelf PRMs scored without any order control. Majority vote sets the floor for Best-of-N. The raw PRMs show the size of the problem. Both run on exactly the same sampled solutions as every other verifier.

The strongest existing scalar PRMs are Qwen2.5-Math-PRM-7B and Skywork-o1-Open-PRM-7B, the models audited in Reward Under Attack, plus VersaPRM. VersaPRM reports a 7.9% gain over majority voting on MMLU-Pro Law, against 1.3% for Qwen2.5-Math-PRM. They are the right competitors because they are what practitioners use today for step scoring and search. For ProcessBench error-localization F1 of these specific scalar PRMs, no reported result was found on Lacuna, so we will measure them ourselves under ProcessBench's protocol.

The strongest verifiers overall are thinking critics, which reason in text before naming the erroneous step. On ProcessBench, o1-mini reports an average F1 of 87.9, QwQ-32B-Preview 71.5, and GPT-4o 61.9 (ProcessBench). We will run QwQ-32B as the open thinking critic on the generated audit, and o1-mini where budget allows, with the same critique prompt across all orderings.

The ablation that isolates our key ingredient is a PRM fine-tuned from the same checkpoint, with the same number of labeled steps and the same tuning budget, on LLM-labeled traces in the VersaPRM style. These come from static math and MMLU-Pro questions, are shown in their original order only, and are labeled by an LLM conditioned on the ground truth. A second ablation keeps our exact generated labels but uses topological order only. Together these separate the contribution of exact contamination-free labels from that of order diversity.

Beating the baselines means three things. First, a flip rate across premise orderings at least halved relative to the best off-the-shelf PRM on generated traces. Second, handoff-error detection within a few points of mid-chain error detection. Third, equal or higher ProcessBench F1 than the static-label ablation, without losing more than a point relative to the base checkpoint. If our PRM matches or exceeds QwQ-32B's critic F1 on the generated audit at a small fraction of its inference cost, that is a practical win; we do not expect to beat o1-mini's static score.

Why not just…?

Why not just average PRM scores over several premise orderings at test time? This is a genuine threat on the audit itself, and we will run it with three and with six orderings. It multiplies verification cost by the number of orderings, which compounds badly with Best-of-N at N=256. It also requires knowing what the premises are and being able to reorder them, which is easy for generated graphs but ill-defined for free-form solutions like those in ProcessBench. If averaging closes the gap at three passes and our trained PRM does not beat it on transfer, we will report averaging as the recommended fix and treat the trained model as a cost reduction only.

Why not just sort premises into dependency order before scoring? On generated problems this canonicalization is trivial and will likely remove premise-order effects entirely, so we include it as an oracle upper bound. It cannot address error-position effects or familiarity effects. It is also not available on natural-language problems, where the dependency structure is the thing being verified. The gap between canonicalization and our trained PRM shows how much order invariance has been learned rather than imposed.

Why not just use a thinking critic, since those already win on ProcessBench? The ProcessBench numbers favor them. But those numbers come from a static benchmark whose problems may be memorized, and they say nothing about order or handoff errors. The faithfulness study suggests that reasoning models themselves under-weight the draft-answer link, so a thinking critic may share the blind spot. The audit tests this directly. If QwQ-32B shows no position or order dependence, the conclusion is that scalar PRMs are the problem and thinking critics should be preferred, and we will say so.

Why not just add permuted orderings to existing PRM training data? This is close to what we do, and we do not claim a special loss. The question is whether order augmentation suffices when the labels are approximate and the source questions are possibly memorized. The second ablation, which uses exact labels in a single order, and the static-label ablation, which uses approximate labels with no order diversity, together answer it. If permuted static data with LLM labels matches our generated data on every metric, the generator's value reduces to auditing rather than training, and we will report that as the finding.

Decisive Checks

The proposal's first claim stands or falls on the audit. If off-the-shelf PRMs show flip rates near zero across premise orderings and detect handoff errors as reliably as mid-chain errors, the hypothesis is wrong and the remaining work is moot. The second claim requires the transfer result. The trained PRM must improve or hold ProcessBench F1 relative to the static-label ablation, or its gains are in-distribution artifacts of the generator.

The third check links the audit to search. Across all verifiers, the drop in Best-of-N accuracy between moderate and large N on held-out generated problems should correlate with measured flip rate. If it does not, order bias is not what drives the large-N degradation, and we will say so.

Finally, the familiarity contrast should show detection on benchmark-styled surfaces exceeding detection on nonsense-noun surfaces of identical graph structure. This is a direct, label-exact measure of how much existing PRMs lean on recognizing familiar problem types rather than checking steps.