Abstract
We consider the problem of learning reward functions and policies from expert demonstrations under unknown dynamics. Our proposed method, Empowerment-regularized Adversarial Inverse Reinforcement Learning (EAIRL), builds on the framework of generative adversarial networks and introduces empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to specific expert demonstrations, leading to generalized behaviors and near-optimal reward recovery. Our method simultaneously learns empowerment through variational information maximization alongside the reward and policy. We evaluate our approach on high-dimensional complex control tasks and challenging transfer learning problems where training and testing environments differ in dynamics or structure. Results demonstrate that EAIRL not only matches expert behavior but significantly outperforms state-of-the-art inverse reinforcement learning (IRL) algorithms in generalization.
Introduction
In Reinforcement Learning (RL), designing a manual reward function for complex tasks like locomotion or navigation is notoriously difficult. Inverse Reinforcement Learning (IRL) aims to circumvent this by learning the reward function directly from expert demonstrations. However, existing IRL methods often suffer from two primary limitations: overfitting to specific expert trajectories and the "State-Only" trap. Many high-performing methods restrict rewards to state-only representations to improve generalization, but this prevents the recovery of true ground-truth rewards that depend on actions (e.g., energy or torque costs).
Methodology: EAIRL
EAIRL introduces a robust framework for learning transferable rewards by incorporating empowerment as a regularizer within an adversarial learning loop.
Empowerment as Regularization
Empowerment ($\Phi$) is an information-theoretic measure that quantifies an agent's potential influence over its future states. By maximizing empowerment, the agent is encouraged to seek "high-utility" states where it maintains the most control. This characteristic of expert behavior prevents the policy from mimicking exact motions (overfitting) and instead focuses on the underlying goal.
Variational Information Maximization
Because calculating empowerment exactly is computationally prohibitive in high-dimensional spaces, EAIRL utilizes a variational lower bound. This allows the agent to learn the inverse model and empowerment network in parallel with the reward and policy, maintaining scalability for complex robotic tasks.
Experimental Setup
The researchers evaluated EAIRL using the OpenAI Gym suite, focusing on both standard benchmarks and difficult transfer learning scenarios:
- Benchmark Tasks: Ant, HalfCheetah, Swimmer, and Pendulum.
- Transfer Learning Benchmarks:
- Crippled Ant: The agent is trained on a standard four-legged ant but tested on an ant with two disabled legs.
- Pointmass-Maze: The agent learns to navigate a maze with a specific passage, which is then blocked and replaced by a new opening in the test environment.
Key Results
EAIRL demonstrates significant improvements in both performance and robustness compared to previous state-of-the-art methods like AIRL and GAIL.
Superior Transfer Learning
In the "Crippled Ant" experiment, EAIRL achieved a score of 348.43, while AIRL collapsed to 46.12. EAIRL's agent successfully adapted its gait to move forward despite the structural change, whereas other agents failed to cope with the altered physics. Similarly, in maze tasks, EAIRL recovered rewards that allowed the agent to find new paths when the original expert route was blocked.
Matching Expert Performance
On standard imitation tasks with consistent dynamics, EAIRL performed comparably to or better than existing methods. In the HalfCheetah task, EAIRL achieved a score of 1870.10, nearing the expert's performance of 2139.83. Furthermore, EAIRL exhibited significantly lower variance across random seeds, indicating a more stable training framework.
Discussion and Insights
The success of EAIRL stems from the realization that experts do not just follow a path; they maintain a state of "flow" and control. By forcing the agent to maximize empowerment, it learns to value the capability to perform a task, leading to rewards that are not tied to specific dynamics. This breaks the previous "disentanglement limit," proving that state-action rewards can be recovered and transferred effectively if properly shaped by empowerment.
Limitations and Future Work
While EAIRL represents a significant step forward, it currently assumes demonstrations come from a single, near-optimal expert. Future research will focus on extending the algorithm to handle "messy" real-world data from multiple experts and learning from sub-optimal demonstrations. Additionally, while scalable, the inclusion of variational networks adds computational complexity compared to simpler behavior cloning methods.
Summary updated