In reinforcement learning, the "reward function" is the primary driver of behavior. However, manually specifying a mathematical reward for complex tasks—like driving a car or folding laundry—is notoriously difficult. Humans often fail to capture every nuance, leading to "reward hacking," where an agent exploits loopholes in a poorly defined objective to maximize its score without actually completing the task.
To solve this, researchers have turned the problem upside down: instead of defining a reward and training an agent to follow it, we observe an expert and infer the reward function they must be optimizing. This field, known as Inverse Reinforcement Learning (IRL) or Apprenticeship Learning, allows agents to learn the "intent" behind actions rather than just mimicking movements.
The Challenge of Manual Reward Design
The fundamental motivation for this research direction is the "engineering burden" of reward design. As Claudia Pérez-D'Arpino and Julie A. Shah point out, creating complex cost functions for collaborative robots is often intractable. When rewards are hand-tuned, they are frequently brittle and tied to specific environment physics.
Justin Fu and Sergey Levine argue in Learning Robust Rewards with Adversarial Inverse Reinforcement Learning that standard imitation learning often fails when the environment changes. If a robot learns to walk by mimicking joint angles, it will fail if its leg length changes. The goal of modern IRL is to recover "disentangled" rewards—objectives that represent the true task (e.g., "move forward") independently of the agent's physical dynamics.
Matching Distributions and Densities
The core mechanism behind many of these algorithms is distribution matching. If an expert's behavior produces a certain distribution of states and actions, the agent should find a reward function that makes its own state-action distribution look identical to the expert's.
Sungjoon Choi and Songhwai Oh pioneered this with Density Matching Reward Learning (DMRL). By using kernel methods to match densities, they bypass the need to model complex environment dynamics entirely. This makes the learning process more stable and computationally efficient.
Adversarial vs Non-Adversarial Approaches
Much of the recent progress has been driven by the "adversarial" framework, inspired by Generative Adversarial Networks (GANs). In algorithms like AIRL (Justin Fu et al.), a discriminator tries to distinguish between expert trajectories and agent trajectories, while the agent (the generator) tries to fool the discriminator. The discriminator's output effectively becomes the learned reward function.
However, adversarial training can be unstable and data-hungry. To address this, researchers have proposed non-adversarial alternatives:
- Primal Wasserstein Imitation Learning (PWIL): Leonard Hussenot and Matthieu Geist introduced PWIL, which minimizes the Wasserstein distance between the agent and expert. By using a simple greedy coupling strategy, they achieved expert-level performance on complex tasks like humanoid running with just a single demonstration.
- Search-Based Methods: Jiafei Lyu and colleagues developed SEABO, which replaces complex neural reward models with a simple nearest-neighbor search using KD-trees. This "offline" approach shows that sometimes, simply staying close to expert data points in state space is more effective than trying to learn a global reward function.
Efficiency and Real-Time Learning
A major hurdle in Apprenticeship Learning is the computational cost. Traditional methods require solving a full Reinforcement Learning problem (an MDP) inside every iteration of the reward-learning loop.
Lior Shani and Tom Zahavy addressed this with Online Apprenticeship Learning (OAL). By framing the problem as a min-max game solved with simultaneous Mirror Descent updates, they allow agents to learn in real-time while interacting with the environment, avoiding the need to solve a full MDP at each step. Zahavy also provided a theoretical bridge, showing that classic apprenticeship methods are actually versions of the Frank-Wolfe optimization algorithm, leading to faster convergence rates.
Scaling to Complexity and Safety
As the field matures, it is moving beyond simple robotics into high-dimensional and multi-agent domains:
- Visual IRL: Aaron David Tucker and Stuart Russell 0001 demonstrated that IRL can scale to high-dimensional video games like Atari by using "pixel-class" autoencoders to learn rewards directly from raw visual input.
- Safety Constraints: In safety-critical scenarios, mimicking an expert isn't enough because the expert might not show what not to do. Junchen Zhao extended Apprenticeship Learning to multi-agent "Markov Games" that incorporate probabilistic safety constraints, ensuring agents learn to be both effective and safe.
- Multi-Objective Tasks: Woo Kyung Kim and Honguk Woo developed ParIRL, which uses diffusion models to reconstruct the "Pareto frontier"—the set of optimal trade-offs between conflicting goals—allowing agents to understand complex human preferences without explicit reward signals.
By shifting the focus from "what the agent should do" to "why the expert did it," these methods provide a more robust, scalable, and human-aligned path for training autonomous systems.