When aligning AI systems using Reinforcement Learning from Human Feedback (RLHF), we often pretend the human teacher is a deterministic oracle. We assume that if trajectory A is objectively better than trajectory B, the human will always select A.
In reality, human feedback is messy, inconsistent, and stochastic. A minor injection of label noise can cause a catastrophic drop in downstream policy performance because standard optimization pipelines mistake human inconsistency for true architectural or behavioral signals (Tien). To build robust AI alignment pipelines, a growing body of research focuses on stochastic preference modeling—treating human choices not as absolute truths, but as noisy, probabilistic samples of an underlying utility landscape.
Why Human Feedback is Inherently Stochastic
Before fixing the noise, we must understand why it occurs. It isn't just random carelessness; it stems from fundamental cognitive and environmental bottlenecks:
- Indistinguishability and Cognitive Friction: When two options are highly similar, humans struggle to perceive a difference. Forcing a choice between nearly identical items creates cognitive friction, leading to erratic, flip-flopping responses (Sui et al.). In complex environments, the visual or behavioral differences between optimal and sub-optimal trajectories are often too subtle for a human to reliably distinguish (Mu et al.).
- Bounded Rationality and Fatigue: Human evaluators suffer from physical fatigue, attention shifts, and ordering effects over long annotation sessions (Li et al.). Furthermore, humans exhibit bounded rationality when asked to optimize high-dimensional problems, leading to inconsistent feedback across dimensions they do not fully comprehend (Mikkola et al.).
- Crowdsourced Disagreement: Modern alignment pipelines scale by gathering feedback from diverse annotator pools. This explicitly violates the assumption of a single rational expert, introducing conflicting internal heuristics, varying expertise levels, and subjective biases into the dataset (An et al.; Yuan et al.).
Smoothing the Noise via Probabilistic Modeling
How do we learn a stable policy from unstable signals? The foundational intuition is to shift from deterministic updates to probabilistic inference.
Instead of treating a binary preference ($A \succ B$) as a hard constraint, algorithms can treat it as a noisy observation of a latent utility function. Wilson et al. demonstrated that by incorporating a probabilistic model of expert behavior, an agent can maintain a robust internal model even when faced with contradictory responses in ambiguous states.
Similarly, Kupcsik et al. utilize a Bayesian approach to reconcile conflicting signals, such as mismatched absolute scores and relative preferences. By modeling the reward function as a distribution—often using Gaussian Processes to explicitly track uncertainty (Gasic et al.)—the system can discount highly uncertain or contradictory feedback, preventing the policy from destabilizing.
Bypassing the Reward Model to Avoid Error Propagation
In standard RLHF, the pipeline is split into two steps: first, train a reward model on human preferences; second, train a policy on that reward model. The danger here is error propagation. If the human feedback is stochastic, the intermediate reward model overfits to the noise, creating proxy rewards that lead to severe policy overoptimization (Kim et al.).
To break this chain, modern algorithms bypass reward inference entirely. Zhang et al. introduced Batched Sequential Action Dueling (BSAD), a model-free algorithm that performs RLHF without ever training an explicit reward model. By directly optimizing the policy through sequential action duels, it avoids the distribution shifts and proxy overoptimization that plague reward-based methods.
Extending this direction, Zhang et al. proposed zeroth-order policy gradient methods designed mathematically to handle noisy, irrational, or contradictory labels in highly stochastic environments, providing provable efficiency without intermediate reward representations.
Structuring Queries to Mitigate Cognitive Friction
Another powerful intuition is to change how we ask questions. If the noise is caused by cognitive friction, we can design query mechanisms that minimize human error by construction.
If humans provide noisy feedback when trajectories are too similar, we can explicitly maximize trajectory distinguishability before asking for input. Mu et al. propose pretraining a diverse library of unsupervised "skills" and strategically asking humans to compare distinct skills rather than raw, chaotic trajectories. This structural change ensures that the human is always presented with clear, distinguishable choices, drastically reducing labeling errors.
Alternatively, we can change the feedback modality itself. Mikkola et al. introduced Projective Preferential Bayesian Optimization (PPBO), which replaces complex pairwise comparisons with 1D projections (such as adjusting a single slider). This reduces the user's cognitive load and provides mathematically denser, less noisy feedback than binary choices, making preference learning highly resilient to human fatigue.