In reinforcement learning (RL), the reward function is the interface through which a designer communicates intentions to an agent. Ideally, this function captures exactly what we want the agent to achieve. However, specifying a dense, pixel-perfect reward function is notoriously difficult. To make the problem tractable, researchers traditionally rely on the linear reward assumption: the premise that the true reward function $R(s,a)$ can be represented as a linear combination of known, hand-crafted features $\phi(s,a)$ parameterized by a weight vector $w$:
$$R(s,a) = w^T \phi(s,a)$$
This single assumption underpins some of the most elegant frameworks in imitation learning, transfer learning, and bandit optimization. Yet, it also introduces a rigid structural bottleneck. If the true human objective involves complex thresholds, logical hierarchies, or non-linear interactions, this linear approximation breaks down, leading to reward misspecification and unintended agent behaviors.
The Elegance of Linearity: Feature Expectations
To understand why the linear reward assumption is so pervasive, we must look at how it simplifies Inverse Reinforcement Learning (IRL) and apprenticeship learning.
When an agent evaluates a policy $\pi$, it computes the expected discounted return. Under a linear reward function, a remarkable algebraic factorization occurs:
$$\mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \right] = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t w^T \phi(s_t, a_t) \right] = w^T \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t \phi(s_t, a_t) \right]$$
The expectation term on the right is known as the feature expectations (or feature occupancies) of the policy, often denoted as $\mu(\pi)$. This vector represents the discounted frequency with which a policy visits specific features in the environment.
Because of this factorization, the expected return of a policy is simply the dot product $w^T \mu(\pi)$. This yields a powerful insight: if an agent can find a policy that matches the expert's feature expectations ($\mu(\pi) = \mu(\pi_E)$), it guarantees identical performance to the expert under the true reward function, even if the true weight vector $w$ remains completely unknown.
This geometric simplification allowed early breakthroughs in robotics. For example, J. Zico Kolter et al. in Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion leveraged this linear decomposition within a hierarchical framework to teach a quadruped robot how to navigate extreme terrain. By framing the problem as matching feature expectations across sub-tasks, the optimization remained stable and computationally feasible.
Similarly, Syed Umar et al. in A Game-Theoretic Approach to Apprenticeship Learning reframed feature matching as a two-player zero-sum game, allowing an agent to efficiently search for policies that minimize the performance gap with the expert. To accelerate this learning process when expert data is scarce, Michal Valko et al. in Semi-Supervised Apprenticeship Learning. demonstrated that unlabeled environment trajectories could be used to shape the underlying feature space, making linear reward recovery significantly more sample-efficient.
Decoupling Dynamics from Desires: Successor Features
The utility of the linear reward assumption extends far beyond imitation learning; it is also the mathematical engine behind zero-shot transfer learning via Successor Features.
Just as the expected return can be factored into a weight vector and feature expectations, the action-value function $Q^{\pi}(s,a)$ can be similarly decomposed:
$$Q^{\pi}(s,a) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t w^T \phi(s_t, a_t) \mid s_0=s, a_0=a \right] = w^T \psi^{\pi}(s,a)$$
Here, $\psi^{\pi}(s,a) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t \phi(s_t, a_t) \mid s_0=s, a_0=a \right]$ represents the Successor Features. They represent the expected future discounted feature occurrences given a starting state and action.
This factorization completely decouples the environment's dynamics (captured by $\psi^{\pi}$) from the agent's immediate goals (captured by $w$). As proven by Andre Barreto et al. in Successor Features for Transfer in Reinforcement Learning., if an environment's transition dynamics remain constant but the task changes, an agent can instantly evaluate an old policy on a brand-new task simply by taking the dot product of its existing successor features with the new reward weight vector $w'$.
This decoupling has proven highly versatile. Angelos Filos et al. in PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning showed that this property allows an agent to learn from unlabelled, reward-free demonstrations of other agents, "borrowing" their structural knowledge of the environment even if their ultimate goals differ. Furthermore, Sunny Amatya et al. in Successor Features for Transfer in Alternating Markov Games extended this framework to multi-agent competitive settings, demonstrating that opponents' strategies can be rapidly identified and countered if their value functions decompose linearly.
Where Linearity Breaks Down
Despite its mathematical convenience, the linear reward assumption is fundamentally limited by the expressivity of its features. If a task's true objective cannot be written as a simple weighted sum, the agent will suffer from representation misspecification.
Consider an autonomous vehicle. A designer might define features for speed ($\phi_1$) and distance to the nearest obstacle ($\phi_2$). A linear reward function $R = w_1\phi_1 + w_2\phi_2$ forces a constant trade-off between the two. If the vehicle is running late (high weight on speed), it might decide that driving at 100 mph is acceptable even if it passes within inches of a pedestrian, because the high positive reward from speed mathematically offsets the negative penalty from proximity.
In reality, human preferences are inherently non-linear and constrained:
- Thresholds: Proximity to an obstacle must remain above a strict safety threshold; once crossed, the penalty should be catastrophic, not linear.
- Target Values: As noted by Erdem Biyik et al. in Active Preference-Based Gaussian Process Regression for Reward Learning, many tasks require reaching a specific target value rather than monotonically maximizing or minimizing a feature.
- Logical Combinations: A task might require an agent to visit point A and point B, but only after collecting a key—a temporal, conditional sequence that cannot be captured by an additive sum of static features.
When the true reward structure is non-linear, forcing a linear model can cause severe alignment failures. The authors of Discovering a set of policies for the worst case reward highlighted that linear assumptions restrict an agent's ability to handle sparse or highly non-linear objectives, limiting the diversity of behaviors the system can safely discover. This risk is particularly acute in safety-critical settings; Jessie Huang et al. in Learning Safe Policies with Expert Guidance pointed out that linear reward functions frequently fail to capture the nuanced, worst-case boundaries required to avoid catastrophic side effects.
Strategies for Richer Reward Models
To bridge the gap between mathematical tractability and real-world complexity, recent research focuses on relaxing or bypassing the linear reward assumption through three main paradigms.
1. Non-Linear Function Approximators
Instead of relying on hand-crafted features and linear weights, modern frameworks substitute deep neural networks or non-parametric models directly into the reward-learning loop. Jonathan Ho et al. in Model-Free Imitation Learning with Policy Optimization bypassed the limitations of linear cost functions by merging apprenticeship learning with generative adversarial frameworks, using neural networks to scale imitation learning to high-dimensional continuous control tasks. For scenarios where data is scarce, Erdem Biyik et al. utilized Gaussian Processes to model complex, non-linear reward surfaces from pairwise human preferences, demonstrating significantly higher data efficiency than rigid linear baselines.
2. Piecewise and Locally Linear Approximations
If a global linear model is too restrictive, a task can be broken down into regions that are individually linear. Bryan Kian Hsiang Low et al. in Inverse Reinforcement Learning with Locally Consistent Reward Functions developed an Expectation-Maximization approach that allows an expert to switch between multiple "locally consistent" linear reward functions mid-trajectory. This captures shifting sub-goals and sequential phases without abandoning the underlying benefits of linear feature matching.
3. Logic and Language-Guided Abstractions
Another approach injects non-linear structure by combining linear features with high-level symbolic side information. Christian Ellis et al. in Task-Guided IRL in POMDPs that Scales integrated Linear Temporal Logic (LTL) specifications into the optimization process, enabling the system to capture complex, time-dependent task constraints that simple weighted sums cannot express.
Alternatively, rather than fixing the feature set $\phi$ in advance, Andi Peng et al. in Adaptive Language-Guided Abstraction from Contrastive Explanations introduced a framework where Large Language Models analyze an agent's failures and autonomously generate code to compute new features. This allows the feature space itself to adaptively grow in response to non-linear or hierarchical task demands.
| Framework | Reward Representation | Key Advantage | Primary Limitation |
|---|---|---|---|
| Classical IRL / Successor Features | Global Linear ($w^T \phi$) | Clean mathematical optimization; instant zero-shot transfer. | Vulnerable to reward misspecification; cannot model thresholds. |
| Adversarial / Deep IRL | Deep Neural Network | High expressivity; captures intricate, high-dimensional behaviors. | Computationally expensive; lacks interpretability and guarantees. |
| Locally Consistent IRL | Mixture of Linear Models | Captures sequential sub-goals and phase transitions. | Requires partitioning trajectories into distinct regimes. |
| Task-Guided / Symbolic IRL | Linear Features + LTL / Language | Captures logical, temporal, and hierarchical constraints. | Relies on the availability of accurate symbolic or language models. |
Key Foundational Work
Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion (2007) — contributed 1 concept elements to this direction.
A Game-Theoretic Approach to Apprenticeship Learning (2007) — contributed 1 concept elements to this direction.
Woodroofe’s one-armed bandit problem revisited (2009) — contributed 1 concept elements to this direction.