When language models are trained with reinforcement learning, a second language model often acts as a judge and scores each intermediate reasoning step. These judges have known biases. They favor longer, more heavily formatted, or authoritative-sounding text, and they rarely admit that such cues influenced them. This proposal argues that the harm from a biased step judge is not spread evenly across the reasoning chain. Many sampled solutions share the same opening steps. A biased score on an early step is therefore copied into every solution that passes through it, and those copies add together into one large gradient push in the same direction. Early steps are also where most multi-step reasoning failures begin. The judge's bias is amplified at exactly the point where it does the most damage.
Two consequences follow, and we plan to test both. First, the gradient contributions from early steps should be heavy-tailed: a few nodes carry disproportionate weight. This is a statistical fingerprint that optimization theory for unbounded gradients knows how to measure. Second, the standard fixes for heavy tails should not remove the bias. Clipping, normalization, and robust averaging over samples all tame magnitude, but the biased component is correlated across every sample that shares the prefix, so these methods leave its direction intact. The fix we propose therefore takes a robust statistic across several differently-presented views of the same step, rather than across sampled solutions. It then normalizes the update separately at each depth so that no single level of the reasoning tree dominates the step.
Core Claim
Consider a group of sampled solutions that share prefixes, forming a tree. Let a prefix node $s$ sit at depth $d$ and be shared by $n_s$ rollouts. Let the judge's score be the true step quality $q(s)$ plus a systematic bias $b(s)$, such as a length preference, and zero-mean noise. Under the prefix-credit reading of Group Relative Policy Optimization (GRPO, a critic-free policy-gradient method that compares rollouts within a group), the bias contributes roughly
$$ g_{\text{bias}} \approx \sum_{s} n_s \, b(s) \, \nabla_\theta \log \pi_\theta(s), $$
where $\pi_\theta$ is the policy. The weight $n_s$ is largest at shallow depth, because every rollout shares its first tokens. The term $b(s)$ is the same for every rollout through $s$, so averaging over those rollouts cannot cancel it.
The argument rests on two prior results. GRPO is Secretly a Process Reward Model showed that GRPO assigns credit to shared prefixes in proportion to $n_s$. It called this "popularity bias" and corrected it with $\lambda$-GRPO, which reweights by $1/n_s$. First-Step Advantage and AgentEval showed that failures concentrate early and propagate: 63% of step failures in agent workflows came from upstream steps. What is new here is combining these with evaluator bias.
We make three predictions. The per-node gradient contributions at shallow depth will show heavier tails as judge bias grows. Robust estimators applied across samples will reduce variance but not the drift that bias induces, for example toward longer outputs or format hacking. Taking a median across bias-neutralizing views of each step, combined with per-depth normalization, will remove both the drift and the heavy tails.
Research Program
The work has three parts: a controlled measurement study, a method, and a transfer to real judges.
The measurement study starts in a setting where judge bias can be dialed exactly. We reuse the depth-four binary-tree environment from the $\lambda$-GRPO paper, where standard GRPO already diverges. We add a synthetic judge whose score is the ground-truth node value plus a tunable bias. The bias either depends on a "verbosity" token or is concentrated at chosen depths.
We then move to GSM8K and MATH-500 with small policies, Qwen2.5-0.5B/1.5B and a 1B Llama, matching the scale of prior work. There the step judge is a real model: Qwen2.5-32B-Instruct, or the reasoning judge QwQ-32B, both studied in Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases. We can still control bias there by injecting length or provenance cues into steps, following the protocol of The Silent Judge.
We log gradient contributions grouped by depth. For each depth we estimate the tail index and kurtosis, using the heavy-tail diagnostics that On Proximal Policy Optimization's Heavy-tailed Gradients applied to continuous control. That paper traced heavy tails to negative advantages and off-policy drift, and it never examined judge-derived advantages or discrete token trees. We also separate each depth's gradient into a variance part and a bias-aligned part. The bias-aligned part is the projection onto the gradient of the bias feature, which is available because we know the injected bias.
The method, view-robust depth-normalized GRPO, changes two things.
First, the per-node reward is taken across views of the step, not across samples. Each prefix step $s$ is scored by the judge under $V$ presentations, $J_1(s), \dots, J_V(s)$. The views include the original step, a length-matched compression, a format-stripped rendering, and a version with provenance or recency cues removed. The per-node reward is
$$ \tilde r(s) = \operatorname{median}_{v \le V} J_v(s). $$
A median is used because it ignores a view that a bias inflates, as long as most views are neutral.
Second, the update is normalized per depth. The gradient is split into per-depth blocks $g_d$, and each block is normalized before summing. This follows the normalized-step argument of From Gradient Clipping to Normalization for Heavy Tailed SGD, which showed that normalized SGD matches clipping without needing a threshold and has parameter-free guarantees under heavy-tailed noise. The same idea appears in the normalized upper-level momentum of Bilevel Optimization under Unbounded Smoothness, which prevented errors from amplifying across levels.
The transfer part asks whether the method still helps when bias is not injected. Here the natural length and format preferences of the real judges are the only bias present.
Baselines
The naive reference is plain GRPO trained on the same step-level judge scores, with a standard KL penalty toward the reference model. It sets the level of bias-induced drift that everything else must reduce.
The strongest existing training-side competitor on reward hacking is Gradient Regularization from Gradient Regularization Mitigates Reward Hacking. It penalizes the gradient norm through finite-difference perturbations and pushes the policy toward flat optima. On GSM8K with Qwen 2.5-0.5B it reports 56.2% accuracy, against 44.2% for the KL baseline. This is the most relevant reported number in the file for our setting. We will rerun it on our setup rather than copy the figure, because our reward comes from a step judge rather than an outcome verifier. The authors themselves note that it can settle into optima that are "flat but wrong." That is precisely the case where a bias shifts the whole landscape.
The strongest credit-assignment competitor is $\lambda$-GRPO. It reports more than a 10% average accuracy gain and wins on 15 of 20 configurations across AIME24, MATH-500, AMC23, Minerva, and OlympiadBench. The file gives no per-benchmark breakdown. Because $\lambda$-GRPO removes the $n_s$ factor, it is the most direct challenger to our mechanism.
The judge-side competitor is PlanJudge, which prompts the judge to write a rubric before scoring. It reports raising DeepSeek-R1 on BiasBench from 65.00 to 97.50. We found no reported result on Lacuna for PlanJudge used as a step-level reward inside RL training.
We will also run BLOCK-GMOM, a robust median-of-means estimator applied across samples, from the heavy-tailed PPO paper.
Two ablations isolate our ingredients: the view median alone, and depth normalization alone. Every method uses the same policy, prompts, judge model, rollout budget, and number of judge calls. The view median uses $V$ judge calls per node, so baselines get $V$ times as many rollouts or judge samples. Every method also gets the same hyperparameter search budget.
We count our method as beating the baselines if it meets three conditions. It must match or exceed GR and $\lambda$-GRPO on final accuracy. It must reduce the bias-aligned gradient component and the resulting length or format drift by a clear margin across three or more seeds. And it must do both in the injected-bias regimes where the baselines degrade.
Why not just…?
Why not just fix the judge with PlanJudge and train normally? This is the most serious alternative. The BiasBench jump shows that prompting can remove much of a judge's bias. However, BiasBench measures pairwise final verdicts, not scores on individual steps. The Silent Judge also found that judges never acknowledged the cues driving their verdicts (0% cue acknowledgment), so a written rubric does not show whether the bias is gone. We test this directly by running plain GRPO with a PlanJudge judge. If that arm matches our method on accuracy and drift, the training-side contribution is unnecessary, and we will report it that way. Our method can also take a PlanJudge judge as input. The comparison that matters is therefore whether the two gains add.
Why not just use $\lambda$-GRPO, since it already removes the $n_s$ weighting? Reweighting by $1/n_s$ removes the amplification factor but leaves $b(s)$ in the update. Each unique path still receives a biased score, which now has equal weight. We therefore predict that $\lambda$-GRPO will reduce heavy tails but not directional drift. The per-depth decomposition in the synthetic tree tests this cleanly. If $\lambda$-GRPO also removes the drift, our central mechanism is wrong.
Why not just clip gradients or use a robust estimator over samples? Clipping and normalization change only the size of the update, not its direction. A median-of-means over rollouts sharing a prefix is taken over copies of the same biased score. From Gradient Clipping to Normalization and the PPO heavy-tail paper both target outliers, not correlated shifts. The BLOCK-GMOM baseline and the depth-normalization-only ablation test this claim directly.
Why not just drop the step judge and use outcome-only verifiable rewards? On GSM8K this is a legitimate option, and we include outcome-only GRPO as a baseline. Local Coherence or Global Validity? found that outcome-only RL produces coherent-looking but invalid chains. Its step-level judgments came from GPT-4o, which was only 57.8% accurate against human labels, so we treat that evidence as suggestive rather than decisive. If outcome-only training matches our method, then step judges are not worth their cost on these tasks, and that is a useful result in its own right.
Proposed Experiments
The first decisive check is in the synthetic tree. We sweep the magnitude of injected bias and the depth where it is placed. We then plot, for each depth, the tail index of the gradient contributions and the bias-aligned drift of the policy, comparing plain GRPO, $\lambda$-GRPO, BLOCK-GMOM, and our method. The mechanism predicts two things. Tails should thicken and drift should grow when bias is concentrated early rather than late. And only the view-median arms should flatten the drift curve.
The second check is on GSM8K and MATH-500 with injected length and provenance cues. We report accuracy, the rate of first-step errors, and the gap between output length and reference length. We also measure a coherence-validity gap on a human-labeled subset, because the GPT-4o judge alone is not accurate enough for this purpose.
The third check uses AIME24, AMC23, Minerva, and OlympiadBench with the naturally biased Qwen2.5-32B and QwQ-32B judges, and no injected cues. This tests whether the effect exists without our intervention.
The claim fails if either of two things happens. Early-depth gradient mass may not grow with judge bias. Or $\lambda$-GRPO combined with a PlanJudge judge may match our method on both accuracy and drift. Either outcome would show that the interaction between bias and prefix sharing is not what drives the damage.
References
- GRPO is Secretly a Process Reward Model
- First-Step Advantage: Importance of Starting Right in Multi-Step Math Reasoning
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- On Proximal Policy Optimization's Heavy-tailed Gradients
- From Gradient Clipping to Normalization for Heavy Tailed SGD
- Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis
- Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains