Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning

Pin Wang, Dapeng Liu, Jiayu Chen, Hanhan Li, Ching-Yao Chan

TL;DR

This paper enhances Adversarial Inverse Reinforcement Learning (AIRL) by incorporating "semantic rewards" to improve its stability and performance in complex, interactive autonomous driving scenarios, specifically lane changes. The augmented AIRL (AugAIRL) successfully learns human-like driving behaviors that outperform existing imitation learning and reinforcement learning methods, achieving performance comparable to expert drivers.

What Problem Does This Paper Address?

Autonomous driving systems need to make complex decisions in dynamic and interactive environments, a task that is notoriously difficult for traditional Reinforcement Learning (RL) due to the challenge of defining an accurate reward function. Imitation Learning (IL) methods, which learn from expert demonstrations, offer a promising alternative. Adversarial Inverse Reinforcement Learning (AIRL) is a state-of-the-art IL technique that can simultaneously learn both a driving policy and the underlying reward function from expert data. However, the original AIRL has primarily been tested in simple, static environments without dynamic interactions. This paper addresses the limitations of AIRL by adapting and augmenting it for the highly interactive and complex task of decision-making in autonomous driving, specifically lane changes, where interactions with other vehicles are crucial and unpredictable.

Methods

The core contribution of this paper is the Augmented Adversarial Inverse Reinforcement Learning (AugAIRL). This method builds upon the existing AIRL framework by introducing semantic rewards. These semantic rewards are designed to be intuitive, meaningful, and easily incorporated into the learning process. Instead of directly adding these semantic rewards to the generator's reward function (which can be subjective and less effective), the authors cleverly integrate them into the discriminator's helper network. This approach ensures that both the discriminator and the generator are updated with the semantic information, leading to improved stability and faster learning.

The paper also details the system design for the decision-making task:

  • Action Space: A high-level, semantically meaningful action space is defined, mimicking human decision-making. This includes actions like "move ahead to gap X and keep current lane" or "move alongside with gap Y and change to target lane," rather than low-level controls like steering and acceleration. This allows for more interpretable and human-like behaviors.
  • State Space: The state representation focuses on features from relevant surrounding vehicles, including their kinematics (speed, acceleration, position, lane ID), to reduce the burden of scene understanding.
  • Policy Optimization: For policy updates, the paper uses the model-free Trust Region Policy Optimization (TRPO) algorithm, as learning a dynamic environment model for driving is deemed too complex.

Datasets

The paper uses expert demonstrations generated from a well-defined rule-based system within a driving simulator. This simulator allows for the creation of diverse and interactive traffic scenarios. The authors mention that this data can be easily replaced with real-world driving data if available. No new benchmarks are introduced; instead, the focus is on evaluating performance within a simulated environment.

Key Insights and Ideas

  • Semantic Rewards in Discriminator: Integrating semantic rewards into the discriminator's helper network, rather than directly shaping the generator's reward, is a key innovation. This approach synchronizes the learning of the discriminator and generator, leading to more stable and effective training.
  • Human-like Decision Making: The high-level action space design allows the agent to learn semantically meaningful decisions, making its behavior more interpretable and human-like. This is crucial for safe and effective interaction with human drivers.
  • "Abort" Behavior: A particularly interesting emergent behavior observed is the agent learning to "abort" a lane change when it detects a risky situation. This is a sophisticated safety mechanism that is often missing in simpler decision-making studies.
  • AIRL's Robustness: The paper highlights that AIRL-based methods (including their augmented version) are more stable than GAIL, likely due to the specific form of the discriminator used.

Key Results

The AugAIRL method consistently outperformed baseline methods (GAIL, AIRL, TRPO, BC+TRPO) across four key evaluation metrics: total reward, success ratio, decision steps, and changing steps.

  • Total Reward: AugAIRL achieved the highest total reward, closely matching expert performance.
  • Success Ratio: The success ratio approached 1.0, indicating a high rate of successful lane changes.
  • Decision and Changing Steps: The number of decision-making and lane-changing steps for AugAIRL was comparable to expert drivers, suggesting human-like maneuvering times and strategies.
  • Stability: AugAIRL demonstrated more stable performance with lower variance compared to other methods, both in training and testing.

Limitations and Future Work

  • Simulation-Based: The current evaluation is conducted in a simulation environment. While realistic, it may not perfectly capture all nuances of real-world driving.
  • Generalization: While the authors believe the approach is generalizable to other imitation learning problems, further validation on different tasks and environments would be beneficial.
  • Future Work: The paper suggests that the learned reward function could be useful for adapting to different environments. Further exploration into how this learned reward function can be leveraged for adaptation and generalization would be a valuable next step. Additionally, exploring more complex interactions and scenarios beyond lane changes could further validate the robustness of AugAIRL.

Summary updated

Abstract

Making decisions in complex driving environments is a challenging task for autonomous agents. Imitation learning methods have great potentials for achieving such a goal. Adversarial Inverse Reinforcement Learning (AIRL) is one of the state-of-art imitation learning methods that can learn both a behavioral policy and a reward function simultaneously, yet it is only demonstrated in simple and static environments where no interactions are introduced. In this paper, we improve and stabilize AIRL’s performance by augmenting it with semantic rewards in the learning framework. Additionally, we adapt the augmented AIRL to a more practical and challenging decision-making task in a highly interactive environment in autonomous driving. The proposed method is compared with four baselines and evaluated by four performance metrics. Simulation results show that the augmented AIRL outperforms all the baseline methods, and its performance is comparable with that of the experts on all of the four metrics.
BibTeX
@inproceedings{wang2021decision,
  author = {Pin Wang and Dapeng Liu and Jiayu Chen and Hanhan Li and Ching-Yao Chan},
  title = {{Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560907},
}
Figure