IL-flOw: Imitation Learning from Observation using Normalizing Flows

Wei-Di Chang, Juan Camilo Gamboa Higuera, Scott Fujimoto, David Meger, Gregory DudekarXiv 2022 · OpenReview

Abstract

We present an algorithm for Inverse Reinforcement Learning (IRL) from expert state observations only. Our approach decouples reward modelling from policy learning, unlike state-of-the-art adversarial methods which require updating the reward model during policy search and are known to be unstable and difficult to optimize. Our method, IL-flOw, recovers the expert policy by modelling state-state transitions, by generating rewards using deep density estimators trained on the demonstration trajectories, avoiding the instability issues of adversarial methods. We demonstrate that using the state transition log-probability density as a reward signal for forward reinforcement learning translates to matching the trajectory distribution of the expert demonstrations, and experimentally show good recovery of the true reward signal as well as state of the art results for imitation from observation on locomotion and robotic continuous control tasks.

Introduction

Imitation Learning (IL) enables agents to acquire complex behaviors by mimicking expert demonstrations. While standard IL often requires state-action pairs, many real-world scenarios only provide state-only observations—a setting known as Imitation Learning from Observation (ILfO). Current state-of-the-art ILfO methods typically rely on adversarial training (similar to GANs), which involves a delicate and often unstable optimization of a discriminator and a policy. IL-flOw addresses these challenges by decoupling reward modeling from policy optimization, utilizing stationary reward signals derived from Normalizing Flows.

Methodology

IL-flOw shifts the paradigm from adversarial competition to distribution matching.

Density Estimation with Normalizing Flows

The core of IL-flOw is the use of Neural Spline Flows to model the probability density of expert state transitions, $p_E(s_{t+1}|s_t)$. Normalizing flows are advantageous as they provide exact log-likelihoods for complex, high-dimensional distributions. Once trained on expert data, the flow model is frozen, providing a stationary reward signal: $r_t = \log p_E(s_{t+1}|s_t)$. This stationarity ensures the agent has a consistent goal throughout training, avoiding the "moving goalpost" problem of adversarial methods.

Noise Conditioning

A significant challenge in density-based rewards is the potential for "spiky" reward landscapes where the signal is only non-zero on the exact expert manifold. To mitigate this, IL-flOw employs a noise-conditioning trick. The flow is trained on expert data augmented with varying levels of noise ($h$).

Visualization of noise conditioning: as the noise level h increases, the density distribution becomes smoother, creating a wider reward "funnel" for the agent.

During reinforcement learning, the agent can effectively "select" the noise level, allowing it to follow a smoother signal when far from the expert path and a more precise signal as it converges. The policy is optimized using Soft Actor-Critic (SAC) to maximize this flow-based reward.

Experimental Results

The method was evaluated on standard MuJoCo benchmarks: Hopper-v2, Walker2d-v2, and HalfCheetah-v2. IL-flOw was compared against strong baselines including f-IRL and MaxEnt IRL across varying numbers of expert trajectories.

Learning curves for IL-flOw compared to baselines across Hopper, Walker2d, and HalfCheetah environments. IL-flOw demonstrates faster convergence and higher average returns.

Key Findings

  • Performance and Stability: IL-flOw consistently outperformed adversarial baselines, achieving near-expert performance even with limited data (e.g., 10 trajectories).
  • Convergence Speed: The stationary nature of the reward signal led to significantly faster and more stable training compared to the moving-target rewards of adversarial methods.
  • Reward Correlation: Analysis showed that the learned log-probability rewards correlate strongly with the true environment rewards, validating the distribution-matching objective as a proxy for task success.

Conclusion

IL-flOw provides a robust framework for ILfO by leveraging the density estimation capabilities of Normalizing Flows. By decoupling reward learning from policy search and introducing noise conditioning, it overcomes the inherent instabilities of adversarial imitation learning. While the current approach assumes approximately invertible dynamics, it demonstrates state-of-the-art performance on complex robotic control tasks. Future work includes extending this approach to high-dimensional pixel observations and exploring environments with highly stochastic dynamics.

Summary updated

Abstract

We present an algorithm for Inverse Reinforcement Learning (IRL) from expert state observations only. Our approach decouples reward modelling from policy learning, unlike state-of-the-art adversarial methods which require updating the reward model during policy search and are known to be unstable and difficult to optimize. Our method, IL-flOw, recovers the expert policy by modelling state-state transitions, by generating rewards using deep density estimators trained on the demonstration trajectories, avoiding the instability issues of adversarial methods. We demonstrate that using the state transition log-probability density as a reward signal for forward reinforcement learning translates to matching the trajectory distribution of the expert demonstrations, and experimentally show good recovery of the true reward signal as well as state of the art results for imitation from observation on locomotion and robotic continuous control tasks.
BibTeX
@misc{chang2022il,
  author = {Wei-Di Chang and Juan Camilo Gamboa Higuera and Scott Fujimoto and David Meger and Gregory Dudek},
  title = {{IL-flOw: Imitation Learning from Observation using Normalizing Flows}},
  year = {2022},
  eprint = {2205.09251},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.09251},
}
Figure