When Judge Panels Amplify Verbosity Bias

Many alignment pipelines no longer trust one AI judge. They ask several language-model judges, or one judge sampled several times, to vote on which of two answers is better, and increasingly they let the judges read each other's rationales before voting again. The hope is that errors average out. This note argues that for one specific error, the preference for longer answers ("verbosity bias"), aggregation and deliberation do the opposite. A weak per-judge lean toward length is a consistent signal, not random noise. Majority-style updates are an error-correcting feedback loop that reinforces whatever the group already leans toward, so they sharpen a weak lean into a near-certain verdict. Opinion-dynamics theory says this sharpening switches on abruptly once the judges' randomness drops below a critical level.

The proposal measures this amplification. It maps where judge sampling temperature, panel size, and number of deliberation rounds tip a panel from mild to near-deterministic length preference. It then tests whether that sharp regime is what causes runaway response length when panel verdicts are used as training labels. The practical payoff is a specific, testable fix: remove each judge's length bias before votes are shared or combined, not after. Because the aggregation step is nonlinear, the standard after-the-fact corrections cannot undo what it did.

Core Claim

Opinion-dynamics research studies agents that repeatedly look at a few neighbours and move toward the local majority, while noise randomly scrambles some opinions. In the 3-Majority model the initial majority survives almost indefinitely when the scrambling probability is below $p_c = 1/3$ and is erased almost instantly above it (Phase Transition of the 3-Majority Dynamics with Uniform Communication Noise, Phase transition of the 3-majority opinion dynamics with noisy interactions). Emergence of Biased Consensus in Multi-Agent LLM Debates carried this picture to debating LLM agents. In their model sampling temperature $T$ is the noise and agent conformity is the coupling. Below a critical temperature, small individual biases lock the group into biased consensus within one or two rounds. The transition softens into a finite-size crossover for groups of 3 to 10, and mixing model families or temperatures smooths it out. That paper studies discrete social, sector, and self-preference biases. It explicitly abstracts away text length, so it never treats length as the bias that gets amplified.

Our claim is that verbosity bias is exactly the kind of weak, shared bias field that these dynamics amplify. Suppose a single judge prefers the longer of two otherwise equivalent answers with probability $p = \tfrac12 + \varepsilon$, where $\varepsilon$ is that judge's length lean. A $k$-judge majority then prefers the longer answer with probability

$$q_k(\varepsilon) = \sum_{j > k/2} \binom{k}{j} p^{j}(1-p)^{k-j},$$

whose slope at $\varepsilon = 0$ grows roughly like $\sqrt{k}$. This is the central-limit growth of a majority's sensitivity to a small consistent bias. Deliberation rounds, in which judges see each other's verdicts, compose this map repeatedly with a conformity pull, and so push $q$ toward 0 or 1 faster than any single vote can.

In a training loop, a policy $\pi$ updated on panel labels drifts toward longer outputs whenever the panel's effective length gain exceeds the restoring force of the policy's anchor to its reference model. Written roughly,

$$\frac{d\,\mathbb{E}_\pi[\ell]}{dt} \propto G(T, k, r) - \beta,$$

where $\ell$ is response length, $G$ is the length gain of a panel at temperature $T$ with $k$ judges and $r$ deliberation rounds, and $\beta$ is the anchor strength. Because $G$ changes steeply with $T$ in the ordered regime, the length trajectory should split sharply as temperature crosses a critical value, rather than growing smoothly as current accounts of over-optimization assume (Rethinking the Role of Proxy Rewards in Language Model Alignment, which documents proxy reward rising while true utility falls).

The new thing is not "judges prefer long answers." That is established: Verbosity Bias in Preference Labeling by Large Language Models quantifies it on HH-RLHF. The new thing is the measured, predictive link between how judges are aggregated and how strongly that bias reaches the training signal, together with the phase boundary this link implies.

Research Program

The work has three stages.

Stage 1: static measurement. We build length-perturbed answer pairs in which substance is held fixed and only length and padding vary. We derive them from RewardBench Chat and Chat Hard items (RewardBench) and HH-RLHF, and add Qasper items where the concise answer is correct and the long one is low-density hedging. Qasper is where Verbosity ≠ Veracity found verbose answers performing 27.61% worse.

On these pairs we measure four quantities:

  • the per-judge lean $\varepsilon$;
  • the panel's effective lean under plain majority vote;
  • the panel's effective lean under soft averaging of vote fractions;
  • the panel's effective lean under deliberation, where judges read one another's verdicts and rationales for one to three rounds.

We sweep temperature from 0 to 1.7, which matches the range used by the biased-consensus study, and panel size from 1 to 7. Judges come from open 7 to 9B models plus one stronger API judge.

Stage 1 tests two predictions. The first is that $q_k$ follows the binomial amplification curve for independent voting. The second is that deliberation adds amplification beyond it, with a steep rise below a critical temperature. We fit the finite-size crossover and locate its midpoint $T^*$.

Stage 2: training loop. We run three to five iterations of online DPO on UltraFeedback prompts with a 7 to 9B policy. Labels come from judge panels placed on either side of $T^*$. We track the mean length trajectory and quality.

We also probe a tension flagged in the research file. Raising temperature to escape biased lock-in may push judges or policies into the uncertain regime where Verbosity ≠ Veracity observed verbosity compensation. If so, a temperature window exists that is neither too cold nor too hot, and we map it rather than assume it.

Stage 3: intervention. The proposed fix is vote-level length calibration. Before any judge's verdict is shared in deliberation or counted, we subtract that judge's own estimated length effect. For the estimate we reuse the locally weighted regression of Post-hoc Reward Calibration, fitted per judge and per temperature on held-out pairs. Error correction then runs on the debiased signal.

The key ablation applies the same calibration only once, to the final panel output. If the amplification story is right, calibrating after aggregation should leave most of the bias in place. The majority map saturates, so the length dependence of the final verdict is no longer the additive, smooth shape the regression assumes.

Baselines

Naive reference. The do-nothing point is a single judge at default temperature and, separately, an uncorrected deliberating panel. The single judge shows the per-judge bias before any aggregation. For GPT-4 on HH-RLHF, Verbosity Bias in Preference Labeling by Large Language Models reports a verbosity-bias metric of 0.328, and 0.428 for GPT-3.5. We will recompute this metric for every judge, so our panel numbers sit on the same scale. For deliberating judge panels themselves, no reported result was found on Lacuna.

Strongest existing correction. This is post-hoc length calibration applied to the panel's final score, exactly as Post-hoc Reward Calibration does for scalar reward models. That paper reports a 3.11 average gain on RewardBench across 33 reward models. It is the strongest competitor because it is training-free, cheap, and aimed at precisely this bias. It will be fitted on the same held-out pairs with the same bandwidth search as our per-judge version. Beating it means a lower effective length lean on the perturbed pairs and a higher RewardBench Chat and Chat Hard accuracy at matched compute. In the training loop, beating it means a bounded length trajectory with no loss in AlpacaEval 2 length-controlled win rate.

Physics-motivated competitor. This is judge heterogeneity: mixed model families and temperatures spread across 1.3 to 1.7. Emergence of Biased Consensus in Multi-Agent LLM Debates showed this smooths lock-in for other biases. It is the natural "just add noise" answer from Direction 1, and it will be run with the same panel sizes and number of judge calls.

Heuristic training-side baselines. Two baselines act on the policy rather than the judges. The first is a hard cap on rationale and answer length, following the 150-word cap of Debate Training Reduces Reward Hacking in RLAIF. The second is length-normalized policy optimization with SimPO (SimPO), given the same tuning budget over its reward scale and margin. SimPO's reported 72.4% length-controlled AlpacaEval 2 win rate is for Gemma-2-9B in a different pipeline. It is a ceiling reference, not a like-for-like target. The closer in-file reference for online RLHF is the 27.00% length-controlled win rate of DPO Meets PPO: Reinforced Token Optimization, against 19.47% for PPO.

Ablations. Calibration after aggregation isolates the key ingredient, which is where calibration happens. Soft averaging versus hard majority, with and without deliberation, isolates which nonlinearity is responsible.

Why not just…?

Why not just average judge scores instead of taking a majority? This is a real threat for panels that do not deliberate. Averaging vote fractions is linear in $p$, so it should not amplify $\varepsilon$ the way the $q_k$ formula does. Stage 1 tests this directly, and if averaging suffices for independent panels we will report exactly that. However, deliberation adds nonlinearity through conformity before any final aggregator is applied. Averaging cannot remove amplification that happened inside the conversation, and the deliberation arm of Stage 1 decides whether this matters in practice.

Why not just cap length? Debate Training Reduces Reward Hacking in RLAIF needed a 150-word cap and reports an expressivity trade-off. Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling shows that length penalties induce a short bias. Our Qasper and Arena-Hard results will show whether capping hurts prompts that need long answers, and whether vote-level calibration avoids that harm.

Why not just calibrate the final panel score? This is the ablation, and the whole proposal turns on it. If after-aggregation calibration matches vote-level calibration, the nonlinearity argument is wrong for practical panel sizes. We would then conclude that existing calibration already suffices.

Why not just raise the judges' temperature? It may work against lock-in, as heterogeneity did for the biases in Emergence of Biased Consensus in Multi-Agent LLM Debates. But it costs verdict accuracy and may trigger uncertainty-driven verbosity. We report accuracy against length lean along the temperature sweep, so any gain from higher temperature is visible together with its cost.

Decisive Experiments

Test one. Is a deliberating panel's effective length lean measurably larger than its members' average lean, and does it rise steeply below some temperature $T^*$ rather than linearly? If the panel lean equals the single-judge lean, the core claim fails.

Test two. In online DPO on UltraFeedback, do length trajectories diverge by panel temperature, with runaway on the cold side of $T^*$ and bounded length on the warm side? Evaluation uses AlpacaEval 2 length-controlled win rate, Arena-Hard style-controlled score, and GSM8K accuracy to check for the reasoning decline noted for SimPO.

Test three. Does vote-level calibration beat after-aggregation calibration on both the perturbed-pair lean and downstream length-controlled quality? Success means a bounded length trajectory, a lower lean than final-score calibration, and no loss relative to the uncalibrated loop on Arena-Hard. Anything less means the phase-transition view adds diagnosis but no new control.