Modern generative models, from Large Language Models (LLMs) to diffusion-based image generators, are rarely deployed in their raw state. Instead, they undergo an alignment phase where a "reward model" (RM) acts as a proxy for human judgment, scoring outputs to steer the generative model toward desirable behaviors. However, a growing body of research reveals a fundamental vulnerability in this pipeline: the generative model does not just learn the task; it internalizes and amplifies the inherent biases, inaccuracies, and systematic errors of the reward model itself.
The Reward Model as a Bottleneck
The alignment process—whether implemented via Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), or test-time search—treats the reward model as an objective oracle. As Ximing Lu et al. note in their work on the QUARK algorithm, the effectiveness of reward-conditioned alignment is strictly constrained by the accuracy and neutrality of this scoring mechanism.
When a generative model is trained to maximize a reward signal, it essentially "absorbs" the reward model's world view. If the reward model systematically prefers certain linguistic styles or aesthetic features, the generative model will replicate them, often at the expense of diversity or objective truth. Kejian Shi demonstrates that this inheritance begins as early as pretraining; when human preferences are incorporated directly into the initial training phase, any systematic blind spots in the reward model are encoded into the model’s foundational weights.
Mechanisms of Inheritance
The propagation of flaws from judge to student occurs across three primary stages of the model lifecycle:
1. Preference Optimization During fine-tuning, methods like DPO or listwise ranking are highly sensitive to the quality of the initial scores. Mingzhang Yin et al. argue that in listwise optimization, reward model inaccuracies are not just inherited but potentially amplified. Similarly, Zhongkai Sun et al. found that in conversational AI, any bias in the reward signal is directly propagated to the target model, reinforcing incorrect behaviors during the alignment process.
2. Test-Time Scaling and Search Even after training, reward models guide the "thinking time" of models during inference. Bowen Zhou et al. highlight that Process Reward Models (PRMs) used to guide reasoning steps often exhibit "over-criticism" (penalizing unconventional but correct steps) or "error neglect." Because the search process converges on paths the RM favors, these biases distort the final output, as also observed by Yangzhen Wu et al..
3. Automated Data Curation When reward models are used to filter or label massive datasets for instruction tuning, they act as a gatekeeper. Alexander Bukharin et al. show that if these scoring models contain inherent biases, the selection process propagates these flaws into the final fine-tuned model, often reducing the robustness of the system on "worst-case" prompts.
The Nature of Propagated Biases
The biases being propagated are rarely random; they are often systematic and structural. Enyu Zhou found that reward model errors are consistent enough that techniques like majority voting fail to correct them. These systematic flaws manifest in several ways:
- Group Fairness: Kefan Song et al. discovered that reward models function as automated judges that systematically undervalue topics or linguistic styles associated with specific academic disciplines or demographic groups.
- Ideological Bias: Suyash Fulay observed that training reward models to prioritize "truthfulness" can inadvertently introduce consistent political biases, which then risk being passed to the generative policy.
- Surface Artifacts: Reward models often fall prey to "reward hacking," where they reward surface-level features like response length rather than actual quality. Jiwoo Hong et al. identify that this is often caused by the excessive growth of hidden state norms in the reward models themselves.
Breaking the Dependency
Recognizing the risks of proxy dependency, researchers are exploring ways to decouple alignment from flawed reward models. Kai Yang et al. propose skipping the reward model entirely by using direct human feedback to fine-tune diffusion models, thereby removing the "middleman" that introduces proxy bias.
Others, like Sungdong Kim and Minjoon Seo, suggest moving away from "black-box" reward models toward "white-box" functions based on interpretable features. By making the reward signal transparent, researchers can better understand—and eventually intercept—the biases before they are inherited by the next generation of AI.