Abstract
We present an algorithm for Inverse Reinforcement Learning (IRL) from expert state observations only. Our approach decouples reward modelling from policy learning, unlike state-of-the-art adversarial methods which require updating the reward model during policy search and are known to be unstable and difficult to optimize. Our method, IL-flOw, recovers the expert policy by modelling state-state transitions, by generating rewards using deep density estimators trained on the demonstration trajectories, avoiding the instability issues of adversarial methods. We demonstrate that using the state transition log-probability density as a reward signal for forward reinforcement learning translates to matching the trajectory distribution of the expert demonstrations, and experimentally show good recovery of the true reward signal as well as state of the art results for imitation from observation on locomotion and robotic continuous control tasks.
Introduction
Imitation Learning (IL) enables agents to acquire complex behaviors by mimicking expert demonstrations. While standard IL often requires state-action pairs, many real-world scenarios only provide state-only observations—a setting known as Imitation Learning from Observation (ILfO). Current state-of-the-art ILfO methods typically rely on adversarial training (similar to GANs), which involves a delicate and often unstable optimization of a discriminator and a policy. IL-flOw addresses these challenges by decoupling reward modeling from policy optimization, utilizing stationary reward signals derived from Normalizing Flows.
Methodology
IL-flOw shifts the paradigm from adversarial competition to distribution matching.
Density Estimation with Normalizing Flows
The core of IL-flOw is the use of Neural Spline Flows to model the probability density of expert state transitions, $p_E(s_{t+1}|s_t)$. Normalizing flows are advantageous as they provide exact log-likelihoods for complex, high-dimensional distributions. Once trained on expert data, the flow model is frozen, providing a stationary reward signal: $r_t = \log p_E(s_{t+1}|s_t)$. This stationarity ensures the agent has a consistent goal throughout training, avoiding the "moving goalpost" problem of adversarial methods.
Noise Conditioning
A significant challenge in density-based rewards is the potential for "spiky" reward landscapes where the signal is only non-zero on the exact expert manifold. To mitigate this, IL-flOw employs a noise-conditioning trick. The flow is trained on expert data augmented with varying levels of noise ($h$).
During reinforcement learning, the agent can effectively "select" the noise level, allowing it to follow a smoother signal when far from the expert path and a more precise signal as it converges. The policy is optimized using Soft Actor-Critic (SAC) to maximize this flow-based reward.
Experimental Results
The method was evaluated on standard MuJoCo benchmarks: Hopper-v2, Walker2d-v2, and HalfCheetah-v2. IL-flOw was compared against strong baselines including f-IRL and MaxEnt IRL across varying numbers of expert trajectories.
Key Findings
- Performance and Stability: IL-flOw consistently outperformed adversarial baselines, achieving near-expert performance even with limited data (e.g., 10 trajectories).
- Convergence Speed: The stationary nature of the reward signal led to significantly faster and more stable training compared to the moving-target rewards of adversarial methods.
- Reward Correlation: Analysis showed that the learned log-probability rewards correlate strongly with the true environment rewards, validating the distribution-matching objective as a proxy for task success.
Conclusion
IL-flOw provides a robust framework for ILfO by leveraging the density estimation capabilities of Normalizing Flows. By decoupling reward learning from policy search and introducing noise conditioning, it overcomes the inherent instabilities of adversarial imitation learning. While the current approach assumes approximately invertible dynamics, it demonstrates state-of-the-art performance on complex robotic control tasks. Future work includes extending this approach to high-dimensional pixel observations and exploring environments with highly stochastic dynamics.
Summary updated