Adversarial Imitation via Variational Inverse Reinforcement Learning

Ahmed Hussain Qureshi, Byron Boots, Michael C. Yip

Overview

Researchers have developed Empowerment-regularized Adversarial Inverse Reinforcement Learning (EAIRL), a novel framework that allows AI agents to learn robust, transferable reward functions from expert demonstrations. By utilizing "empowerment"—an information-theoretic measure of an agent’s control over its environment—as a regularizer, the model prevents overfitting to specific expert paths. This enables the agent to successfully recover rewards that remain effective even when the agent's physical structure or the environment's layout changes.

The Challenge of Inverse Reinforcement Learning

In traditional Reinforcement Learning (RL), an agent is guided by a hand-crafted reward function (e.g., "+1 for reaching the goal"). However, designing these functions for complex, nuanced tasks like robotic surgery or autonomous driving is incredibly difficult. Inverse Reinforcement Learning (IRL) attempts to solve this by "reverse-engineering" the reward function directly from expert demonstrations.

Despite its promise, IRL faces two major hurdles:

  1. Overfitting: Agents frequently memorize the expert's specific movements rather than understanding the underlying goal. Consequently, they fail when the environment changes even slightly.
  2. The "State-Only" Limitation: Recent state-of-the-art methods, such as Adversarial IRL (AIRL), struggle to recover rewards that depend on both the agent's actions and its state. In the real world, actions like "applying high torque" often carry an inherent penalty, but previous models couldn't capture this effectively while keeping the learned reward transferable to new situations.

The EAIRL Approach: Empowerment as a Secret Sauce

The authors introduce EAIRL, building upon the Generative Adversarial Network (GAN) framework. In this setup, a "Generator" (the policy) attempts to mimic the expert, while a "Discriminator" tries to distinguish between expert trajectories and the agent's trajectories.

The key innovation is the integration of Empowerment:

  • What is Empowerment? It is a measure of how much influence an agent has over its future state. An empowered agent is positioned such that its actions can lead to a wide variety of reachable future states.
  • Variational Information Maximization: Because calculating empowerment exactly is computationally expensive, the authors use a variational lower bound to estimate it efficiently in parallel with reward and policy learning.
  • Regularization: Empowerment acts as a crucial regularizer. Instead of blindly following the expert's exact footsteps, the agent is intrinsically motivated to seek states where it maintains high control. This prevents the policy from collapsing into a narrow set of memorized behaviors and fosters the learning of a generalized reward function.

Datasets and Transfer Learning

To prove EAIRL's robustness, the researchers evaluated it on several high-dimensional continuous control tasks from the OpenAI Gym (MuJoCo) benchmark, including standard tasks like Ant, HalfCheetah, Swimmer, and Pendulum.

Crucially, they tested the learned rewards in challenging Transfer Learning Benchmarks where the training and testing environments differ significantly:

Normal and Crippled Ant Environments
  • Crippled-Ant: The reward is learned on a normal four-legged ant but tested on an ant with two disabled legs. The agent must understand the core goal of moving forward and adapt its gait accordingly.
Pointmass Maze Transfer Environments
  • Pointmass-Maze: The reward is learned in a maze with a passage on the left, but tested in a modified maze where the passage has moved to the right. The agent must navigate the new structure without relying on memorized paths.

Key Insights and Results

  • Superior Transfer Learning: EAIRL dramatically outperformed existing methods in transfer scenarios. In the "Crippled-Ant" task, EAIRL achieved a score of 348.43, successfully teaching the crippled ant to change its gait to move forward. In contrast, the previous state-of-the-art (AIRL) collapsed to a score of just 46.12.
  • Adaptable Navigation: In the maze transfer task, EAIRL recovered a reward that successfully guided the agent to the goal even when the original path was blocked and a new one opened. Competing methods failed to generalize to the new layout.
  • Breaking the State-Only Constraint: Unlike previous methods that ignored actions to make rewards transferable, EAIRL successfully handles state-action rewards. This is vital for tasks where the manner of movement (e.g., avoiding jerky motions) is as important as the destination.
  • Disentanglement: By using an empowerment-based potential function, the learned reward becomes "disentangled" from specific environment dynamics. This explains why a reward learned on a healthy ant remains applicable to a crippled one.
  • Competitive Imitation: On standard tasks where the environment remains static, EAIRL performed as well as or better than GAIL and AIRL, demonstrating that its added robustness does not come at the cost of baseline performance.

Limitations and Future Work

While highly effective, the current EAIRL model assumes all demonstrations come from a single, optimal expert. Real-world data is often noisy or sourced from multiple individuals with varying styles. A promising future direction is adapting the algorithm to learn from sub-optimal or "rank-amateur" demonstrations—extracting the core goal even when the expert makes mistakes. Additionally, while the variational approach is efficient, calculating empowerment still introduces computational overhead compared to simpler imitation methods.

Summary updated

Abstract

We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The results show that our proposed method not only learns near-optimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.
BibTeX
@inproceedings{qureshi2019adversarial,
  author = {Ahmed Hussain Qureshi and Byron Boots and Michael C. Yip},
  title = {{Adversarial Imitation via Variational Inverse Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=uMIlZSkkeCQ},
}
Figure