Standard Inverse Reinforcement Learning (IRL) relies on a heavy assumption: the agent being observed is an expert. To figure out what a robot or a human wants, we typically wait until they have mastered a task and then analyze their optimal behavior. But in the real world, expertise is a finish line, not a starting point. We often observe agents while they are still stumbling, adapting, and improving.
A growing body of research suggests that we don't need to wait for perfection. Instead of looking at what an agent does at equilibrium, we can look at how an agent changes. By treating the learning process itself as the signal, we can recover reward functions faster and more robustly than by watching experts alone.
The Learning Trajectory as a Signal
In traditional IRL, we look for a reward function that makes the observed behavior look optimal. If an agent is still learning, however, their current behavior is by definition sub-optimal, which leads standard IRL to recover "broken" rewards.
The key insight from Giorgia Ramponi, Gianluca Drappo, and Marcello Restelli is that while a learner's policy is sub-optimal, the direction of their improvement is highly informative. In their work on LOGEL, they demonstrate that if we assume an agent follows a gradient-based update rule, we can treat their policy change as a vector pointing toward the reward.
If you see an agent move from Policy A to Policy B, and you know they are trying to maximize some reward, the "distance" and "direction" between A and B reveal the underlying objective. Remarkably, Ramponi et al. show that a single learning step can be enough to recover a reward function with high accuracy, provided the observer has a good estimate of the learner's policy gradient.
Inverting the Update Rule
This idea of "inverting" the learning process extends beyond simple gradients. In multi-agent environments, where multiple agents are learning and reacting to one another, the problem becomes a moving target.
Mine Melodi Caliskan, Francesco Chini, and Setareh Maghsudi developed the MA-LfL framework to handle these dynamics. They found that the log-ratio of an agent’s consecutive policies—essentially the mathematical difference in how an agent prioritizes actions over time—serves as a rich signal for identifying objectives.
The intuition here is that the "improvement" between two versions of a policy filters out the noise of sub-optimal exploration. By analyzing the divergence between what the agent did yesterday and what it does today, an observer can isolate the specific reward components that drove that change. As the learner's performance improves, the quality of the recovered reward scales logarithmically, making this approach increasingly powerful as the agent nears expertise.
Bridging Imitation and Reinforcement Learning
This "inversion" logic also solves a practical problem in robotics: how to fine-tune a robot after it has learned from demonstrations. Usually, we use Behavioral Cloning (BC) to copy an expert, but if we then try to improve that robot using Reinforcement Learning (RL), the robot often "unlearns" the expert's nuances because the RL reward doesn't perfectly match the expert's internal goals.
Joe Watson, Sandy H. Huang, and Nicolas Heess addressed this with Coherent Soft Imitation Learning (CSIL). Instead of guessing a reward, they mathematically invert the entropy-regularized Bellman equation.
By treating a pre-trained BC policy as the "optimal" starting point of a learning process, they derive a shaped reward function for which that policy is already the solution. This ensures that when the robot begins its own RL training, it starts from a state of mathematical coherence—the reward it is chasing is perfectly aligned with the demonstrations it just saw.
Why It Works
The shift from "Learning from Experts" to "Learning from Learners" works because it leverages the causal mechanism of improvement.
- Density of Information: An expert's static policy tells you where the peak of the mountain is. A learner's update tells you the slope of the mountain. The slope is often easier to identify from limited data.
- Robustness to Sub-optimality: By modeling the update rule (like gradient descent), we explicitly account for the fact that the agent makes mistakes. We aren't trying to match their actions; we are trying to match their intent to improve.
- Mathematical Inversion: Whether through policy gradients or log-ratios, the learning step is a function of the reward. If we know the function and we see the step, we can solve for the reward.
By watching the journey rather than just the destination, these methods allow AI systems to understand the goals of their peers and teachers much earlier in the learning process.