Adversarial Imitation via Variational Inverse Reinforcement Learning

Ahmed Hussain Qureshi, Michael C. Yip

Abstract

We consider the problem of learning reward functions and policies from expert demonstrations under unknown dynamics. Our proposed method, Empowerment-regularized Adversarial Inverse Reinforcement Learning (EAIRL), builds on the framework of generative adversarial networks and introduces empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to specific expert demonstrations, leading to generalized behaviors and near-optimal reward recovery. Our method simultaneously learns empowerment through variational information maximization alongside the reward and policy. We evaluate our approach on high-dimensional complex control tasks and challenging transfer learning problems where training and testing environments differ in dynamics or structure. Results demonstrate that EAIRL not only matches expert behavior but significantly outperforms state-of-the-art inverse reinforcement learning (IRL) algorithms in generalization.

Introduction

In Reinforcement Learning (RL), designing a manual reward function for complex tasks like locomotion or navigation is notoriously difficult. Inverse Reinforcement Learning (IRL) aims to circumvent this by learning the reward function directly from expert demonstrations. However, existing IRL methods often suffer from two primary limitations: overfitting to specific expert trajectories and the "State-Only" trap. Many high-performing methods restrict rewards to state-only representations to improve generalization, but this prevents the recovery of true ground-truth rewards that depend on actions (e.g., energy or torque costs).

Methodology: EAIRL

EAIRL introduces a robust framework for learning transferable rewards by incorporating empowerment as a regularizer within an adversarial learning loop.

Empowerment as Regularization

Empowerment ($\Phi$) is an information-theoretic measure that quantifies an agent's potential influence over its future states. By maximizing empowerment, the agent is encouraged to seek "high-utility" states where it maintains the most control. This characteristic of expert behavior prevents the policy from mimicking exact motions (overfitting) and instead focuses on the underlying goal.

Variational Information Maximization

Because calculating empowerment exactly is computationally prohibitive in high-dimensional spaces, EAIRL utilizes a variational lower bound. This allows the agent to learn the inverse model and empowerment network in parallel with the reward and policy, maintaining scalability for complex robotic tasks.

Experimental Setup

The researchers evaluated EAIRL using the OpenAI Gym suite, focusing on both standard benchmarks and difficult transfer learning scenarios:

  • Benchmark Tasks: Ant, HalfCheetah, Swimmer, and Pendulum.
  • Transfer Learning Benchmarks:
    • Crippled Ant: The agent is trained on a standard four-legged ant but tested on an ant with two disabled legs.
  • Pointmass-Maze: The agent learns to navigate a maze with a specific passage, which is then blocked and replaced by a new opening in the test environment.
Transfer learning benchmarks: The Crippled Ant (left) and the Pointmass-Maze (right) environments used to test reward generalization.

Key Results

EAIRL demonstrates significant improvements in both performance and robustness compared to previous state-of-the-art methods like AIRL and GAIL.

Superior Transfer Learning

In the "Crippled Ant" experiment, EAIRL achieved a score of 348.43, while AIRL collapsed to 46.12. EAIRL's agent successfully adapted its gait to move forward despite the structural change, whereas other agents failed to cope with the altered physics. Similarly, in maze tasks, EAIRL recovered rewards that allowed the agent to find new paths when the original expert route was blocked.

Matching Expert Performance

On standard imitation tasks with consistent dynamics, EAIRL performed comparably to or better than existing methods. In the HalfCheetah task, EAIRL achieved a score of 1870.10, nearing the expert's performance of 2139.83. Furthermore, EAIRL exhibited significantly lower variance across random seeds, indicating a more stable training framework.

Discussion and Insights

The success of EAIRL stems from the realization that experts do not just follow a path; they maintain a state of "flow" and control. By forcing the agent to maximize empowerment, it learns to value the capability to perform a task, leading to rewards that are not tied to specific dynamics. This breaks the previous "disentanglement limit," proving that state-action rewards can be recovered and transferred effectively if properly shaped by empowerment.

Limitations and Future Work

While EAIRL represents a significant step forward, it currently assumes demonstrations come from a single, near-optimal expert. Future research will focus on extending the algorithm to handle "messy" real-world data from multiple experts and learning from sub-optimal demonstrations. Additionally, while scalable, the inclusion of variational networks adds computational complexity compared to simpler behavior cloning methods.

Summary updated

Abstract

We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The results show that our proposed method not only learns near-optimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.
BibTeX
@inproceedings{qureshi2019adversarial,
  author = {Ahmed Hussain Qureshi and Michael C. Yip},
  title = {{Adversarial Imitation via Variational Inverse Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=uMIlZSkkeCQ},
}
Figure