Keeping the Judge On-Distribution: Audited Online Adaptation of Reward Judges in Offline-to-Online Preference Learning

When a language model is first aligned on a fixed set of preference data and then allowed to generate and learn from its own samples, its outputs drift away from anything the scoring model was trained on. This two-stage recipe is the offline-to-online transition: start from logged data, then improve by interacting. The scoring model is usually a small reward model or an LLM used as a judge, meaning a language model asked to say which of two answers is better. Current pipelines update the policy and keep the judge frozen. The judge is therefore grading text it has never seen, and the policy is rewarded for finding the judge's blind spots. We propose to treat the judge as a second learner that also undergoes an offline-to-online transition. A small local judge is checked against an expensive strong judge on a tiny "audit" budget of the policy's current outputs. The size of their disagreement then decides when the local judge is retrained and which anchor it is held to while it is retrained: its own offline initialization, or the original offline preference data.

The claim is testable and specific. We expect the policy to start gaming the reward at about the point where the local judge's accuracy on fresh policy samples falls. We expect that accuracy drop to be measurable with a few hundred strong-judge calls per round. And at a fixed strong-judge budget, we expect audit-triggered, on-distribution judge updates to keep the reward honest better than three alternatives: a frozen judge, a judge refreshed on a fixed schedule, or routing uncertain pairs to the strong judge without ever updating the local one.

What Is Known and What Is Missing

The motivating evidence comes from online preference learning. RLHF Workflow moves LLaMA-3-8B from offline DPO (Direct Preference Optimization, which fits the policy directly to preference pairs) to iterative online DPO scored by a frozen proxy reward model. It raises the AlpacaEval-2 length-controlled win rate from 22.5% to 31.3%. It also finds that a length penalty is needed because the frozen reward model rewards verbosity. BPO shows that the policy's trust region should be anchored to the model that generated the current rollouts, not to the initial fine-tuned model. Both papers fix the policy side of the distribution shift and leave the evaluator untouched. Value-Incentivized Preference Optimization unifies offline pessimism and online optimism, but it assumes stationary preference labels.

Two papers constrain the design.

  • Reward hacking generalizes fast. Debate Training Reduces Reward Hacking in RLAIF shows that training against a frozen, weaker LLM judge collapses true accuracy, and that hacking generalizes across tasks more easily than real competence. Its debate fix was only tested on verifiable math.
  • Judges hide their biases. The Silent Judge shows that GPT-4o shifts verdicts by up to 30% on recency cues and 18% on provenance cues, with a 0% rate of acknowledging those cues. A judge's rationale therefore cannot tell us whether it is being gamed. We need an external audit.

The closest method is Ask a Strong LLM Judge when Your Reward Model is Uncertain.

  • What it does. It routes pairs that the local preference model is unsure about to DeepSeek-R1. On RM-Bench-hard this lifts accuracy from 44.9% to 81.9%. On Arena-Hard it reaches a 29.8% win rate, against 28.5% for random routing.
  • What it leaves out. The strong judge's verdicts are used once and discarded, so the local model never improves.
  • Why that matters. Its uncertainty estimate is a distance from the local model's own training data. It cannot catch confident errors on inputs that look familiar but have been subtly gamed.

Our proposal differs in two ways. The audit is a random sample of current rollouts rather than an uncertainty-selected one. And the audited labels are written back into the local judge.

RUBRIC-ARM alternates judge and rubric updates, but only on offline data with a single-pass DPO policy. Our setting has the policy and judge co-adapting online.

The method we reuse comes from offline-to-online reinforcement learning (RL).

  • Which anchor to keep. The Three Regimes of Offline-to-Online Reinforcement Learning finds that the right thing to stay close to depends on whether the pretrained policy beats the offline data. Staying near the policy wins in 75% of "superior" settings, and staying near the data wins in 83% of "inferior" settings. A simple return comparison picks the right method 71% of the time.
  • How to weight old and new data. The Adaptive Replay Buffer weights replayed data by how likely the current policy is to produce it.

We transfer both ideas from the policy to the judge.

Core Claim

Let $J_0$ be the local judge after offline training on preference data $\mathcal{D}$. At online round $t$ the policy is $\pi_t$. Let $a_t$ be the local judge's agreement with the strong judge on a random audit set of $m$ pairs sampled from $\pi_t$. Let $a_{\mathcal{D}}$ be the same agreement measured on held-out offline pairs.

We define the audit gap as

$$g_t = a_{\mathcal{D}} - a_t .$$

Positive $g_t$ means the judge has fallen behind the policy. Because $a_t$ is an average of $m$ agreement indicators, its standard error is about $\sqrt{a_t(1-a_t)/m}$. So $m \approx 300$ resolves drops of roughly five points.

Our claim has two parts:

  1. Rising $g_t$ comes before the divergence between local-judge reward and strong-judge reward. It can therefore serve as an early warning.
  2. Retraining the local judge whenever $g_t$ crosses a threshold works best under two conditions. The audited pairs must be mixed with $\mathcal{D}$ using on-policyness weights. And the anchor must follow a judge-side version of the three-regime test. If $J_0$ agrees with the strong judge more than the offline labels do on current samples, we regularize toward $J_0$. If it agrees less, we replay $\mathcal{D}$ more heavily.

Research Program

Diagnostic study. We first run the frozen-judge pipeline of RLHF Workflow and log $g_t$ every round. We also log two measures of gaming:

  • the hacking gap: the local judge's reward minus the strong judge's reward on the same rollouts;
  • a shortcut-exploitation rate: the fraction of rollouts containing authority, provenance, or recency framing, and how much the local judge's verdict changes when that framing is removed. This is the Silent Judge verdict-shift protocol applied to the policy's own outputs.

This study answers whether judge decay predicts hacking, and by how many rounds it leads.

Method. We then build the audited loop. The policy update stays as in iterative DPO, optionally with BPO's behavior-anchored reference. Each round, $m$ random pairs from $\pi_t$ go to the strong judge. When $g_t$ exceeds a threshold, the local judge is fine-tuned on a buffer that mixes audited pairs with $\mathcal{D}$. Each pair is weighted by the current policy's geometric-mean token likelihood, as in the Adaptive Replay Buffer. Separately from that weighting, the anchor choice decides whether the judge is pulled toward $J_0$ or toward $\mathcal{D}$.

Judge models. The local judge is a Bradley-Terry reward model, which turns pairwise preferences into a scalar score. As a second condition we use a generative judge initialized in the style of Think-J. Running both tests whether reasoning judges decay more slowly.

Baselines

Do nothing. The naive reference is offline DPO on the same preference data with no online phase. RLHF Workflow reports 22.5% AlpacaEval-2 length-controlled win rate for this setting.

Frozen local judge. The main existing method is the frozen-reward-model iterative DPO of RLHF Workflow, including its length penalty. Its reported result is a 31.3% AlpacaEval-2 length-controlled win rate and 8.46 on MT-Bench with LLaMA-3-8B. We will reproduce it on the same SFT model and prompt set. Beating it means:

  • a statistically higher length-controlled win rate under an independent judge not used in training;
  • a hacking gap that stays bounded where the baseline's grows;
  • an equal or lower average response length.

Uncertainty routing. The strongest competitor that also uses a strong judge is uncertainty routing as in Ask a Strong LLM Judge. It reports 29.8% versus 28.5% random routing on Arena-Hard. We give it exactly the same total number of strong-judge calls as our method, the same policy optimizer, and the same tuning budget for its threshold as for ours. Beating it means a higher Arena-Hard win rate and a lower final shortcut-exploitation rate at equal calls.

Fixed-schedule refresh. This baseline retrains the local judge every $k$ rounds on uniformly mixed strong-judge labels of current samples, again at an equal call budget. It is the most important control, because it isolates whether the audit trigger and anchor choice add anything beyond "relabel recent data."

Ablations. Three ablations isolate the key ingredients:

  • remove the on-policyness weights;
  • fix the anchor to $J_0$ only, and separately to $\mathcal{D}$ only;
  • spend the audit budget on uncertainty-selected pairs instead of random ones.

Ceiling. A frozen strong judge used on every pair gives a cost-unconstrained ceiling. It is reported with its call count.

Why not just…?

Why not just use a frozen frontier judge for every rollout? Debate Training Reduces Reward Hacking in RLAIF shows that frozen LLM judges are themselves hacked. The Silent Judge shows that frontier judges carry large unacknowledged cue biases, and the routing paper notes that per-rollout strong-judge calls are prohibitively expensive. We include this as a ceiling. If it is not hacked in our setting and our method matches it at a fraction of the calls, that is the result. If it is hacked, the audit loop cannot help either, because the audit relies on the strong judge. We would report that as a limit of the approach.

Why not just retrain the local judge on fresh strong-judge labels every few rounds? This is the real threat. It may capture most of the benefit, since the main ingredient is putting on-policy labels into the judge. The fixed-schedule refresh baseline at equal budget settles it. If it matches our method on win rate and hacking gap, the contribution shrinks to the diagnostic finding that $g_t$ predicts hacking, and we will say so. We expect the audit trigger to win mainly in runs where drift is bursty, for example after the policy discovers a new shortcut. We will report results per run, not only averages.

Why not just add a length penalty or other fixed shaping? RLHF Workflow already needs one. It removes one known bias but not the provenance and recency shortcuts documented by the Silent Judge, which a fixed penalty cannot anticipate. Our shortcut-exploitation measurements under the penalized baseline will show directly whether such shortcuts appear.

Why not just route uncertain pairs, as existing work does? Distance-based uncertainty flags inputs that look unfamiliar. Gamed outputs that look ordinary but contain authority phrasing may be scored confidently and wrongly. The random-audit versus uncertainty-audit ablation tests exactly this.

Proposed Experiments

Setup. The policy is LLaMA-3-8B following the RLHF Workflow SFT recipe. Prompts and offline pairs come from UltraFeedback. The strong judge is a large open reasoning model. Final evaluation uses AlpacaEval 2.0 length-controlled win rate, Arena-Hard, MT-Bench, and IFEval, which provides a verifiable instruction-following check that no judge can be gamed on. All win rates are re-scored by a judge excluded from training. Local-judge quality is tracked on RewardBench 2 and RM-Bench to detect forgetting of general criteria.

Decisive checks. We test four things:

  1. Early warning. Over five or more seeds, does $g_t$ rise at least one round before the hacking gap diverges?
  2. Fixed-schedule refresh. At equal strong-judge calls, does the audited judge beat fixed-schedule refresh and uncertainty routing on the independent win rate and on IFEval?
  3. Anchor choice. Does the judge-side regime test pick the better anchor more often than chance? We score it against the result of running both anchors.
  4. Shortcut gaming. Do provenance and authority phrasings increase under frozen judges, and does that increase stop under audited updates?

Kill conditions. A negative answer on check 1 kills the trigger idea. A negative answer on check 2 against fixed refresh reduces the work to a measurement paper.