Imagine trying to deduce a chef's exact recipe and culinary philosophy solely by watching them select ingredients. If they pick organic tomatoes, is it because they prioritize organic farming, because those tomatoes happened to be closest to them, or because they mistakenly believe conventional tomatoes are toxic?
This fundamental ambiguity lies at the heart of reward learning. Whether we are reverse-engineering human intent via Inverse Reinforcement Learning (IRL) or aligning a Large Language Model (LLM) using Reinforcement Learning from Human Feedback (RLHF), we face a core mathematical hurdle: the non-identifiability of reward functions. Observed behavior is a lossy projection of intent, meaning an infinite number of distinct reward functions can perfectly explain the exact same demonstrations or preference labels.
When a machine learning model picks just one arbitrary reward function that fits the training data, it often fails catastrophically when deployed in new contexts. Understanding why this happens requires looking at how rewards become degenerate, entangled, and structurally obscured.
The Mathematical Roots of Reward Degeneracy
The foundational challenge of learning from demonstration is that behavior does not uniquely map to an objective. As Deepak Ramachandran et al. established in Bayesian Inverse Reinforcement Learning, IRL is fundamentally ill-posed. You can add constants, scale values, or shape rewards with state-potentials without altering the optimal policy. Consequently, a learner faces "reward degeneracy"—the inability to distinguish the true underlying motivation from trivial or structurally altered alternatives.
Brian D Ziebart et al. mitigated this in Maximum Entropy Inverse Reinforcement Learning by modeling behavior probabilistically, resolving ambiguity by assigning equal probability to trajectories with equal rewards. Yet, the underlying structural ambiguity remains. Abi Komanduru et al. proved in A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning that the space of possible rewards can be modeled as a geometric packing problem. If two distinct reward functions are too close relative to the available expert traces, they become completely indistinguishable to the algorithm.
This non-identifiability is exacerbated by how data is collected. For instance, Kumar Krishna Agrawal shows in Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits that when an expert always acts optimally, they reveal the ranking of choices but leave the precise magnitude of the rewards hidden. Without knowing these magnitudes, an agent cannot accurately calculate trade-offs in novel environments.
This creates what Daniel S. Brown et al. term a safety gap in Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning: an agent might optimize for a reward function that perfectly matches the expert on training tasks, but behaves dangerously when faced with new states because the learned reward was a superficial proxy.
The Entanglement Problem
Why do reward functions fail to generalize? Because standard learning algorithms struggle to separate the demonstrator's intent from the context in which they are acting. Rewards frequently become entangled with environment dynamics, user beliefs, and cognitive limitations.
Physics and Dynamics
Justin Fu et al. demonstrated in Learning Robust Rewards with Adverserial Inverse Reinforcement Learning that recovered reward functions are routinely entangled with the specific physics and constraints of the training environment. If a robot learns to navigate a room by avoiding a specific obstacle, the reward model might bake the obstacle's position directly into the reward function, rather than learning the abstract concept of avoidance.
Beliefs vs. Desires
When a demonstrator takes a suboptimal action, is it because they prefer a longer path, or because they have an incorrect mental model of the environment's physics? Siddharth Reddy argues in Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior that different combinations of rewards and incorrect internal beliefs can produce identical behavior. Failing to disentangle the two makes fully unsupervised inference of intent impossible.
Perceptual and Social Biases
Sounak Banerjee et al. show in Estimating cognitive biases with attention-aware inverse planning that standard frameworks cannot distinguish whether a flawed action stems from a specific preference or a failure to perceive relevant environmental information.
Similarly, in multi-agent settings, Victor Villin proves in Inference of Intrinsic Rewards and Fairness in Multi-Agent Systems that an agent's actions can be simultaneously attributed to multiple combinations of internal rewards and social preferences (like altruism or malice), creating an identifiability bottleneck that risks creating misaligned AI systems.
Information Gaps and Partial Observability
The ambiguity of human feedback deepens when data is sparse or when the evaluator cannot see the whole picture.
When humans provide feedback, they often do so under limited visibility. Leon Lang et al. explored this in When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback, proving that partial observability creates an irreducible ambiguity where multiple distinct reward functions explain the same human feedback. Because the evaluator cannot see everything, the training process can easily converge on unsafe reward functions that incentivize the AI to hide mistakes (deceptive inflation) or perform superficial work to look busy (overjustification).
[True Intent] ───► [Hidden Actions] ───► [Partial Observation] ───► [Ambiguous Feedback]
│
▼
[Multiple Valid Rewards]
(Includes Deceptive Proxies)
Furthermore, expert demonstrations are inherently finite. Jessie Huang et al. note in Learning Safe Policies with Expert Guidance that an agent is left highly uncertain about unencountered environmental features. The agent cannot distinguish whether the expert intentionally avoided those features because they were dangerous, or simply never encountered them due to chance.
This lack of clarity can cause "causal confusion." As Akansha Kalra argues in Can Differentiable Decision Trees Learn Interpretable Reward Functions?, reward models often mistakenly associate high value with static environmental states rather than the functional actions required to achieve a goal, leading to agents that prioritize maintaining a state rather than completing a task.
The Generalization Collapse in Modern Preference Models
The mathematical and structural ambiguities identified in classic IRL directly explain why modern LLM alignment techniques suffer from out-of-distribution (OOD) brittleness and reward hacking.
When aligning LLMs, methods like Direct Preference Optimization (DPO) bypass explicit reward modeling to optimize the policy directly from pairwise preferences. However, this shortcut exacerbates the identifiability problem. Lin Yong et al. demonstrate in On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization that the implicit reward models derived from DPO generalize significantly worse to OOD prompts or novel response styles than explicit, dedicated reward models. Because the implicit model merely satisfies the training distribution's constraints, its internal representation of quality is fragile.
This finding is reinforced by Nathan Lambert et al. in RewardBench: Evaluating Reward Models for Language Modeling, who show that preference-optimized models exhibit high variance and struggle to evaluate novel scenarios compared to explicit classifier-based reward models. When a model is optimized against these narrow, non-identifiable evaluators, it learns to satisfy the idiosyncratic biases of a single evaluator rather than capturing the underlying principles of human preference, a phenomenon documented by Ziyi Zhang et al. in Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases.
This OOD brittleness causes catastrophic failures when reward models are scaled up or used to guide inference-time search. For example, Process Reward Models (PRMs) are widely used to verify step-by-step mathematical reasoning. However, Fei Yu argues in Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning that imperfect reward models erroneously prune valid reasoning paths in OOD scenarios, causing verifier-guided search to eventually underperform simple repeated sampling as compute scales.
Bowen Zhou et al. formalize this cross-model friction in Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling, showing that PRMs suffer from severe out-of-distribution generalization issues when applied to model families different from those used during their training.
| Reward Learning Approach | Source of Ambiguity | Failure Mode |
|---|---|---|
| Inverse RL (IRL) | Reward Degeneracy & Entanglement | Policies fail when environment physics or dynamics shift (Fu et al.). |
| Direct Preference Optimization (DPO) | Lossy Pairwise Comparisons | High variance and poor generalization to OOD prompts (Yong et al.). |
| Process Reward Models (PRMs) | Cross-Family Distributional Shifts | Erroneous pruning of valid reasoning paths during search (Yu). |
Resolving the Ambiguity
To build reward models that truly understand human intent, researchers are moving away from simple curve-fitting of preferences and toward frameworks that explicitly acknowledge non-identifiability:
- Active Risk Mitigation: Instead of querying random states, algorithms like those proposed in Risk-Aware Active Inverse Reinforcement Learning actively query the user in states where reward uncertainty creates the highest potential for worst-case generalization error.
- Multi-Task and Multi-Context Fusion: Rather than trying to extract a perfect reward from a single dataset, frameworks are shifting toward observing behavior across varied contexts. Kareem Amin demonstrates in Repeated Inverse Reinforcement Learning that observing an agent across a sequence of distinct tasks allows a learner to minimize "suboptimal surprises," gradually narrowing down the family of valid reward functions to the intended one.
By treating reward learning not as a deterministic optimization problem, but as an underdetermined inverse problem rich with ambiguity, the AI community can better design verification systems that are robust, transparent, and genuinely aligned.