Does the Random Head Teach Reward Models Their Shortcuts?

Reward models and trained judges, the language models that score or rank other models' answers, pick up shallow shortcuts. They prefer longer answers, answers that cite sources (even invented ones), and answers labeled as written by an "expert." They also never admit to using these cues. Most proposed fixes act on the data: swap answer orders, drop references, add counterexamples. This note asks whether part of the problem comes from how training starts. A standard reward model is built by bolting a newly initialized scoring layer, the "head," onto a pretrained language model and training everything at once. For the first few hundred steps that head outputs noise, and the large, noisy error signal flows back into the whole network. At that moment the cheapest features for reducing loss are length and surface style, not correctness.

Gradual unfreezing is an old remedy for exactly this early phase. You train the head first with the backbone frozen, then release layers one at a time from the top down. The proposal is to test, mechanistically and on public judge benchmarks, whether it reduces shortcut absorption and protects hard-reasoning judgments in reward models. The test uses a built-in control: generative judges, which reuse the pretrained vocabulary layer and have no random head, should benefit much less.

Core Claim

The claim is conditional and falsifiable. Take judges trained with a randomly initialized scalar head, which is how most reward models on RewardBench are built. For these judges, the degree to which the final model's scores respond to non-substantive cues is largely set during the early high-loss steps. Head-first, top-down unfreezing should therefore lower that sensitivity and raise accuracy on the "Chat Hard" and "Reasoning" subsets, relative to training all layers from step zero, at equal compute. For generative judges trained to emit verdict text, the effect should shrink toward zero, because nothing in their output path starts random.

If both kinds of judge improve equally, the "gradient shock" story is wrong and any gain is ordinary regularization. If neither improves, the shortcut problem really does live in the data, as the augmentation-based judge papers implicitly assume.

Where this sits in prior work

Gradual unfreezing comes from ULMFiT. That work fine-tuned LSTM classifiers by thawing layers top-down with lower learning rates for earlier layers, and reported 18–24% error reductions. An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models argued that isolating newly initialized components until they settle prevents damage to pretrained layers. FUN with Fisher supplies the method and the diagnostic we reuse. It schedules adapter unfreezing by Fisher information, a measure of how sensitive the loss is to each parameter, and it showed that scheduled unfreezing produces an early "hill" in the trace of the Fisher matrix whose width tracks generalization, worth about 2 points on average in cross-lingual QA and NLI. None of these papers touches evaluators. In none of them is the outcome of interest which features the model learns to rely on; they only measure accuracy.

On the judge side, three papers train open judges end to end on distilled labels and fight bias only through data. JudgeLM fine-tunes LLaMA/Vicuna on 105K GPT-4 judgments with swap augmentation and reference drop. PandaLM filters position-inconsistent pairs. Prometheus 2 merges separately trained scoring and ranking models. These papers motivate the work, because they leave the optimization schedule untouched. Two auditing papers establish the gap. Justice or Prejudice? catalogs 12 biases through perturbation robustness. The Silent Judge shows verdict shifts of up to +30% from recency labels and +18% from "EXPERT" labels, with a 0% rate of the judge acknowledging the cue. Both are black-box audits of already-trained judges and say nothing about when during training the sensitivity forms.

Revisiting Catastrophic Forgetting in Large Language Model Tuning constrains the design. It shows that flatter-minimum optimizers (SAM) and weight interpolation (Wise-FT) reduce forgetting by 7.01% and 5.82% respectively. Any unfreezing gain must therefore beat these cheap optimizer-level fixes, not just plain fine-tuning.

We deliberately do not pursue the more obvious pitch, "gradual unfreezing reduces forgetting in judges." Freezing lower layers, LoRA, SAM, and Wise-FT are all simpler and already known to reduce forgetting. Unfreezing is unlikely to beat them on that axis. The random-head mechanism is the part only unfreezing speaks to.

Research Program

The core object is a 7B decoder backbone of the kind used in Prometheus 2 (Mistral-7B), trained into two judge types on the same preference data.

The first type is a scalar reward model with a new linear head, trained on a pairwise loss. The training data are JudgeLM-100K pairs converted to preferences by the GPT-4 score difference, with ties dropped.

The second type is a generative judge trained on the same pairs to emit scores and a rationale in JudgeLM format.

For each judge type we compare training schedules at a fixed token budget. The schedules range from all layers trainable from step zero, through head warmup, to full ULMFiT-style top-down unfreezing and Fisher-ranked unfreezing. The details are given under Baselines.

The analysis is where the proposal earns its keep. At checkpoints spaced logarithmically through the first 10% of training, and at the end, we measure two things. The first is cue sensitivity: how much the model's score for a response moves when we inject a non-substantive cue while holding content fixed. We use the CALM perturbation types for verbosity and fabricated citations, and the Silent Judge provenance and recency labels. The second is per-layer Fisher trace, as in FUN. The mechanistic prediction is that under all-at-once training, cue sensitivity rises sharply in the first few hundred steps, concentrated in updates to middle layers, and then plateaus instead of receding.

We test the causal direction with a transplant experiment. We take the all-at-once model's early checkpoint, which already carries the absorbed shortcuts, and finish training it under the gradual schedule, and we do the reverse as well. If the early phase is what fixes the shortcuts, the final bias should follow the early schedule, not the late one.

Baselines

The naive reference point is standard reward-model training. A randomly initialized head sits on the backbone, all parameters are trainable from step zero, and a single tuned learning rate is used with cosine decay. This is how most reward models are trained, and it is the condition in which the gradient-shock hypothesis says shortcuts form. Every other condition matches its data, token count, and hardware. Each condition gets the same learning-rate sweep of five values, selected on a held-out split of JudgeLM-100K and never on the bias or RewardBench test sets. Three seeds are run per condition.

The key ablation is head warmup followed by unfreezing everything at once. We train only the head for the same number of steps the gradual schedule spends on its head-only stage, then release all layers simultaneously. This isolates the proposal's central ingredient. If head warmup captures the full benefit, the layer-by-layer order is irrelevant and the story reduces to "don't backpropagate from a random head." That is still a useful finding, but a smaller one.

The strongest optimizer-level competitor combines SAM with Wise-FT interpolation between the base and fine-tuned weights. This is the pair that Revisiting Catastrophic Forgetting found most effective against forgetting. It gets the same sweep budget, with its interpolation coefficient chosen on the held-out split. The strongest parameter-efficient competitor is LoRA on all layers from step zero, alongside FUN's Fisher-scheduled LoRA unfreezing.

The strongest data-level competitor is counterfactual augmentation. We add training pairs in which a response is padded or given a fake citation and labeled as not preferred, with the extra examples counted inside the fixed token budget.

For external reference we report the leaderboard figure RewardBench gives. RewardBench reports an overall score of 89.0 for ArmoRM-Llama3-8B, but that model uses different data and a different backbone, so it is a ceiling marker and not a controlled baseline. The file contains no RewardBench category scores for our training configuration, so no reported result was found on Lacuna for a JudgeLM-100K-trained reward model on RewardBench, and that number will be produced here. For CALM there is likewise no reported robustness rate for open fine-tuned judges on Lacuna.

Beating the baselines means the following. Averaged over three seeds, gradual unfreezing must lower cue-induced verdict shift on held-out cue types relative to both naive training and head-warmup-only. It must do so by more than the seed spread, while matching or exceeding them on RewardBench overall and improving Chat Hard or Reasoning.

Why not just…?

Why not just warm up the head and then fine-tune everything? This is the most serious threat, because it removes the random-noise phase with one extra stage and no layer scheduling. We do not argue against it in advance. It is the central ablation. The layer-wise schedule is justified only if top-down release adds a measurable reduction in cue sensitivity beyond head warmup. The FUN result, that simultaneous release fails to produce the early Fisher hill even in adapters, is suggestive but comes from classification in a different domain.

Why not just use LoRA, which already keeps the backbone frozen? A frozen backbone limits forgetting but does not stop the adapters from learning shortcut features from noisy early gradients. FUN with Fisher found that scheduled adapter unfreezing beat all-at-once adapter training by 2 points on average and by 8 F1 on Thai XQuAD. We include both all-at-once and scheduled LoRA, so the experiment shows whether scheduling matters inside parameter-efficient training for judges too.

Why not just add counterfactual training pairs for each known bias? For a bias you know about, this will likely work as well or better, and we expect it to win on verbosity when it is trained against verbosity. The argument for a schedule-level fix is that it is bias-agnostic. So the decisive comparison uses held-out cues. We augment only against length and citations, then evaluate on provenance and recency labels from The Silent Judge. If augmentation generalizes across cue types as well as unfreezing does, the scheduling claim loses its practical case.

Why not just use SAM or Wise-FT? These target sharp minima and weight drift, and they demonstrably reduce forgetting. They do not target which features the model learns in its first steps. If SAM plus Wise-FT matches gradual unfreezing on held-out cue sensitivity, the mechanism is generic flatness rather than protection from the random head. That outcome would falsify our specific story.

Proposed Experiments

The main experiment trains the reward-model and generative-judge versions of each schedule on JudgeLM-100K. We evaluate on four things. The first is RewardBench's four categories. The second is CALM perturbations for verbosity, fabricated citations, and sentiment, with factual items drawn from GSM8K, MATH, and ScienceQA and scored by robustness rate. The third is Silent Judge-style provenance and recency cues on ELI5 and LitBench, scored by verdict shift rate. The fourth is agreement on the JudgeLM validation set and the PandaLM human test set. For context on the PandaLM test set, PandaLM reports 66.87% accuracy for PandaLM-70B and 66.47% for GPT-4. For the generative arm we also report JudgeLM's position delta bias, where JudgeLM reports 13.11 for the 7B model and 1.63 for the 33B model. We also report cue acknowledgment rate in the generative judge's rationales.

Three checks decide the question. The first is the interaction test: the gap between gradual and naive training in held-out cue sensitivity should be substantially larger for the scalar reward model than for the generative judge. If it is not, the random-head mechanism is rejected. The second is the timing test: under naive training, cue sensitivity measured at the early checkpoints should already predict final cue sensitivity, and the checkpoint-transplant runs should show that final bias follows the early schedule. The third is the held-out-cue comparison against targeted augmentation, which determines whether the method matters in practice as well as mechanistically.

A secondary check tests whether scoring-first output ordering behaves like a random head. We retrain the generative judge with JudgeLM's score-first format versus rationale-first. If score-first judges benefit more from gradual unfreezing, the effect tracks where a high-variance loss concentrates on a few new output decisions, and not merely the presence of a literally random layer.