Automated checkers that grade the intermediate steps of a language model's math solution are now used both to measure "reasoning quality" and to train models. They are fooled by how a step reads rather than whether it is right. A step-level grader, called a process reward model (PRM), will score a fluent but false derivation highly, raise its score when an optimized string of junk tokens is appended, and reward filler such as "wait, let me think." This proposal tests a simple explanation for much of that failure. The grader sees too much. It reads each step inside a long, persuasive narrative, so the prose, the earlier steps, and any injected cues all reach the judgment. Our method strips that away. We rewrite each step into a small, self-contained claim: the stated premises and the one assertion the step makes, in a fixed notation with no surrounding text. The claim is then checked in isolation, by symbolic execution where possible and by a language model judge otherwise.
The bet connects two observations that have so far been studied separately. On the reasoning side, splitting a problem into short, isolated sub-tasks protects a model from errors and drift that build up in long contexts. On the evaluation side, graders inherit artifacts from whatever they condition on. If both hold, isolating the step should do two things. It should make step verification robust to stylistic and adversarial manipulation. It should also make the measured error rate depend much less on which grader is used, which is the real problem with proxy-based evaluation. The expected price is that isolation cannot see errors that are only visible globally, such as a valid step that leads nowhere. We plan to measure that price rather than hide it.
Core Claim
A large share of process-grader failure is context contamination, not inability to check a local inference. Concretely, take the same judge model. Moved from full-context grading to grading context-stripped canonical claims, it should do three things. It should keep most of its earliest-error detection F1 on clean ProcessBench. It should lose far less F1 under semantics-preserving style perturbations, reflection-marker injection, and adversarial suffixes. Different judges should also agree much more with each other about which steps are wrong. The third prediction is the classifier-dependence claim. Process-soundness numbers reported in the literature are partly properties of the grader. Isolation should shrink the grader-specific component.
What Prior Work Establishes and Leaves Open
Three papers motivate the gap by showing that graders respond to surface form.
- Reward Under Attack perturbed and attacked PRMs on PRM-BiasBench, AIME and MATH-500. It found that PRMs act as "reasoning-style detectors": scores move less than 0.1 under rephrasing, logically corrupted steps go unpenalized, optimized 100-token suffixes inflate rewards on invalid traces up to 4×, and closed-loop GRPO reaches reward above 0.9 while accuracy stays below 4%. Its suggested defenses were left unimplemented.
- Assessing Judging Bias in Large Reasoning Models showed that reasoning-model judges are swayed by injected cues, including "superficial reflection" phrases. Their thinking traces sometimes start out correct and then rationalize the bias. Prompt-based mitigations reduced factual-task bias by only up to 14% (system prompt) and 16% (self-reflection). That paper grades whole solutions, not steps.
- ProcessBench supplies the ground truth we need: expert-annotated earliest error steps on 3,400 cases. It shows that on Omni-MATH, 51.8% of correct-answer solutions contain reasoning errors, so outcome checks cannot substitute for step checks.
Two lines of work try to make graders better while still grading steps in context.
- The Bidirectional Process Reward Model adds a reversed-order stream so early steps are judged with hindsight. It reports a 37.7% relative F1 gain in step-level error detection.
- AURORA trains on solutions from seven generators with ensemble soft labels and answer-conditioned "reverse verification." It reaches 74.3 F1 on ProcessBench.
Both give the grader more context. We give it less. Neither tested robustness to attack.
Three papers supply method ingredients.
- When Splitting Makes Stronger argues that shorter, isolated sub-task contexts shield attention from intermediate error pollution. We transfer this from solving to verifying.
- Evaluating Intermediate Reasoning of Code-Assisted LLMs found a decision tree on syntax-tree features (81%) beats an LLM judge (73%) at classifying code logic. This is evidence that stripping prose in favor of structure helps graders.
- Multi-TAG cross-checks steps with Python and WolframAlpha. We reuse this to route computational claims to execution.
Evaluating Step-by-Step Reasoning through Symbolic Verification (LMLP) is the closest relative: it translates steps into predicates and enforces a chain rule. It requires a pre-built knowledge base; we do not.
Two papers constrain the design.
- Evaluating Step-by-step Reasoning Traces: A Survey separates validity (is the step sound given its premises?) from utility (does it help reach the answer?). Isolation can only test validity, so we report the two separately.
- Can LLMs Always Solve Easy Problems if They Can Solve Harder Ones? found that even strong models fail some easy sub-problems: GPT-4 Turbo is 92.2% consistent, and 7B models are often 50–70%. An isolated claim is an easy problem, so a small judge will be noisy on it. This is why we use execution where possible and judges of at least 32B.
Research Program
The pipeline has three stages.
- Extraction. An extractor model reads the problem and the solution up to and including step $k$. It outputs a canonical record for step $k$: the premises the step relies on (problem givens plus specific earlier conclusions, by reference) and the single new claim. The claim is written in a fixed notation, either an equation, inequality, substitution or set statement, or a one-sentence natural-language assertion when no formal form exists. The output format has no free-text field, so filler, reflection markers and appended token strings have nowhere to go unless they change the claim itself.
- Checking. Claims typed as computational (arithmetic, algebraic identity, substitution, solving) are checked with a symbolic algebra system. All other claims go to a language model judge. The judge sees only the premises and the claim and answers "follows / does not follow."
- Scoring. The first failing step is the predicted earliest error. A separate, optional utility pass looks at the claim sequence (not the prose) to flag steps that are valid but irrelevant. It is reported separately.
The method has an obvious weak point that a reviewer will see immediately, and we treat it as a first-class object of study. The extractor can silently repair an error, for example writing the correct value when the step contained a slip. It can also invent an error. We measure extraction fidelity in two ways. On ProcessBench steps with a known error, we check automatically whether the numeric values and relations in the extracted claim still contain the annotated mistake. We also hand-audit 300 sampled steps for repair, invention and premise omission. Because the extractor itself reads the full context, it could be manipulated too, so all robustness experiments include attacks aimed at the entire pipeline, not only the judge.
The analysis target is a matrix of judge × condition × perturbation. Its rows are a set of judges spanning families and sizes: the 7B PRMs and two open reasoning models. Its columns are full-context and isolated grading. The perturbation types follow Reward Under Attack: rephrasing, verbosity, reflection-marker insertion, and optimized suffixes. From this matrix we estimate how much variance in measured step-error rate is explained by judge identity in each condition.
Baselines
Naive references. The naive reference is outcome-only checking: mark a solution as flawless if its final answer is correct. Given ProcessBench's 51.8% figure on Omni-MATH, this sets a floor that any process method must clear on hard tiers. A second naive reference is the same judge grading full context with a plain prompt.
Strongest specialized graders. These are the trained 7B PRMs: Qwen2.5-Math-PRM-7B, reported at 73.5 ProcessBench F1, and Universal-PRM-7B from AURORA, reported at 74.3 F1. BiPRM (The Bidirectional Process Reward Model) is retrained on the same data as its unidirectional counterpart, since its 37.7% gain is relative to its own baselines.
Strongest critic. The strongest critic-style competitor is a reasoning model grading in full context. ProcessBench reports 87.9 average F1 for o1-mini and 71.5 for QwQ-32B-Preview, with GPT-4o at 61.9. The fairest head-to-head is the same open model, QwQ-32B or a comparable open reasoning model, as judge in both full-context and isolated modes. This holds weights, decoding settings and prompt-tuning budget fixed, and changes only what the judge sees. Isolation makes several calls per solution, so we also give the full-context judge self-consistency votes up to the same total token budget.
Prompted debiasing. We include the system-prompt and self-reflection mitigations from Assessing Judging Bias in Large Reasoning Models, applied to the full-context judge.
Post-hoc calibration. We apply a LOWESS-style correction in the manner of Post-hoc Reward Calibration, regressing step scores on step length and filler-token counts.
Ensemble. We include an ensemble of the two 7B PRMs, which Reward Under Attack found to have complementary blind spots.
Ablations. Four conditions isolate the key ingredient:
- (a) full context with filler removed by pattern matching;
- (b) isolated premises and claim, but in the step's original wording;
- (c) the full canonical claim;
- (d) canonical claims with symbolic execution switched off.
Comparing (a), (b) and (c) separates the effect of "removing narrative context" from "removing surface wording." Condition (d) shows how much depends on the execution tool rather than the isolation.
What counts as winning. On clean ProcessBench, isolation must come within 3 F1 of the same judge in full context. Under each perturbation family, it must lose at most half as much F1 as the best full-context baseline. Under the adaptive pipeline attack, it must hold reward inflation on invalid traces well below the 4× that Reward Under Attack reports for PRMs. No published number exists for any of these baselines under ProcessBench-with-perturbations; we found no reported result on Lacuna for perturbed ProcessBench, so we establish these numbers ourselves.
Why not just…?
Why not just use the strongest reasoning model as a full-context critic? This is the real threat on clean data: o1-mini's 87.9 F1 on ProcessBench may beat anything we build, and we do not expect to exceed it on clean inputs. The evidence that it is not enough comes from Assessing Judging Bias in Large Reasoning Models. Reasoning-model judges are moved by reflection phrases and injected cues, and they rationalize those cues in their thinking traces, so strength on clean inputs does not imply robustness. The same-judge comparison settles this directly. If the full-context reasoning judge keeps its F1 under our perturbations and pipeline attacks, isolation adds nothing beyond cost, and we would report that.
Why not just strip filler phrases or calibrate scores post hoc? For literal markers such as "wait, let me think," pattern removal may work about as well as isolation, and ablation (a) is there to show it. It cannot handle the main finding of Reward Under Attack: fluent but mathematically false steps get high scores, and optimized suffixes are not a fixed vocabulary. LOWESS calibration further assumes the biased feature is independent of quality, which Post-hoc Reward Calibration itself flags. Step length is plausibly correlated with legitimate derivation depth. If (a) matches (c) on every perturbation family, the extraction stage is unnecessary, and we would drop it.
Why not just ensemble PRMs or use a bidirectional PRM? Ensembling exploits the complementary blind spots of Skywork-PRM and Qwen-PRM documented in Reward Under Attack, and it is a genuine competitor. But both members still read the full persuasive context. A suffix optimized against the ensemble, which we will run, tests whether that shared input is the weak point. BiPRM adds hindsight, but the reversed stream reads the same text and could be a second surface to attack. That paper did not test this, and our attack suite does.
Why not just formally verify steps? Translating to Lean or to a knowledge base, as LMLP (Evaluating Step-by-Step Reasoning through Symbolic Verification) does, gives guarantees where it applies. It needs a complete formalization, which competition-level natural-language solutions rarely allow. Our pipeline uses symbolic checking exactly where it is cheap and falls back to an isolated judge elsewhere. Ablation (d) reports how many ProcessBench steps actually get executable claims, which bounds how far formal methods alone could go.
Proposed Experiments and Decisive Checks
The primary benchmark is ProcessBench, reported per tier (GSM8K, MATH, OlympiadBench, Omni-MATH) with its earliest-error F1. We build a perturbed copy using the transformations of Reward Under Attack: rephrasing, verbosity, and reflection-marker insertion at the position of the erroneous step. Optimized suffixes are generated in two ways. Transferred suffixes are optimized against white-box 7B PRMs. Adaptive suffixes are optimized against the full isolation pipeline by black-box search, with the extractor in the loop. We also evaluate on PRM-BiasBench and on the UniversalBench long-CoT problems from AURORA, to test whether extraction holds up on long, meandering traces.
Check 1: the robustness gap. Isolation should keep F1 close to full context on clean data while losing much less under perturbation. If the robustness gap does not appear for the same judge, the context-contamination hypothesis is wrong.
Check 2: judge dependence. Across at least five judges, we compute the variance in per-tier error-rate estimates and the pairwise agreement on flagged steps (Cohen's kappa), in both conditions. The classifier-dependence claim holds only if isolation substantially raises inter-judge agreement on clean data, not just under attack.
Check 3: extraction fidelity. If silent repair exceeds roughly 10% of annotated erroneous steps, the method is unsound as stated. Any robustness it shows could simply come from rewriting errors away.
Check 4: what isolation misses. We break down misses by ProcessBench error, separating local-validity errors from strategy or relevance errors. This tells readers exactly which error types isolation gives up.
Check 5: closed-loop training (scaled-down). We replicate the closed-loop training test of Reward Under Attack at small scale. A 1.5B–7B policy is trained with GRPO on MATH training problems under three rewards: a 7B PRM, the full-context judge, and the isolated-claim reward, each combined with final-answer correctness. We track the gap between reward and held-out MATH500 accuracy, and the share of reward gain explained by style features. The claim survives only if the isolated reward does not show the reward-up, accuracy-down divergence that the PRM shows.
References
- Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models
- Assessing Judging Bias in Large Reasoning Models: An Empirical Study
- ProcessBench: Identifying Process Errors in Mathematical Reasoning
- The Bidirectional Process Reward Model
- AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
- A Toolbox, Not a Hammer - Multi-TAG: Scaling Math Reasoning with Multi-Tool Aggregation
- Evaluating Step-by-Step Reasoning through Symbolic Verification
- Evaluating Step-by-step Reasoning Traces: A Survey
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- Post-hoc Reward Calibration: A Case Study on Length Bias