Adversarial Inverse Reinforcement Learning for Decision Making in Autonomous Driving

Pin Wang, Dapeng Liu, Jiayu Chen, Ching-Yao Chan

TL;DR

This paper enhances Adversarial Inverse Reinforcement Learning (AIRL) by incorporating "semantic rewards" to improve its performance and stability in complex, interactive autonomous driving scenarios, specifically lane changes. The augmented AIRL method achieves expert-level performance in simulations, demonstrating its potential for learning human-like driving behaviors.

What Problem Does This Paper Address?

Autonomous driving requires agents to make complex decisions in dynamic and interactive environments, a task that traditional Reinforcement Learning (RL) struggles with due to the difficulty of hand-crafting reward functions. Imitation Learning (IL) methods, which learn from expert demonstrations, offer a promising alternative. Adversarial Inverse Reinforcement Learning (AIRL) is a state-of-the-art IL technique that can learn both a driving policy and a reward function simultaneously. However, existing AIRL has primarily been tested in simple, static environments without dynamic interactions. This paper addresses the challenge of applying and improving AIRL for the highly interactive and nuanced task of decision-making in autonomous driving, particularly lane changes, where human-like behavior and safety are paramount.

Methods

The core contribution of this paper is the augmentation of the existing AIRL framework. The authors introduce "semantic rewards" into the learning process. These rewards are designed to be intuitive, sparse, and semantically meaningful, incorporating domain knowledge about driving objectives like success, collision avoidance, margin invasion, and lateral movement. Crucially, these semantic reward terms are added to the discriminator's helper network, allowing them to influence both the discriminator and, consequently, the generator's learning. This approach differs from simple reward shaping, which only affects the generator, by ensuring that the discriminator is also updated to account for these semantic objectives. This synchronization is key to improving training stability and performance. The augmented AIRL uses a model-free policy optimization method, Trust Region Policy Optimization (TRPO), for policy updates, as learning a full environment model for complex driving is deemed too challenging.

Datasets

The paper utilizes expert demonstrations generated from a well-defined rule-based system within a driving simulator. This system has full access to the behavioral models of surrounding vehicles, allowing it to generate realistic and diverse expert driving scenarios for training. No new datasets were introduced; instead, the focus was on adapting existing methods to a simulated driving environment that mimics real-world complexities.

Key Insights and Ideas

  • Semantic Rewards for Stability: Simply adding semantic rewards to the generator's objective (reward shaping) only marginally improved performance. The key insight was to integrate these semantic rewards into the discriminator's learning, leading to more stable and effective training.
  • Human-like "Abort" Behavior: The proposed system, with its high-level action space, enabled the agent to learn an "abort" behavior. This means the agent can intelligently decide to cancel a lane change if it detects a risky situation, a nuanced and human-like response often missing in simpler decision-making studies.
  • Bridging the Gap to Experts: The augmented AIRL method not only outperforms baseline imitation learning and RL methods but also achieves performance comparable to expert drivers across multiple evaluation metrics. This suggests that learning from demonstrations with augmented rewards can effectively capture complex, expert-level decision-making strategies.

Key Results

The augmented AIRL (AugAIRL) method demonstrated superior performance compared to GAIL, AIRL, TRPO, and BC+TRPO in simulation. Specifically:

  • Total Reward: AugAIRL achieved the highest total reward, closely matching expert values.
  • Success Ratio: AugAIRL's success ratio approached the expert level of 1.0 faster than other methods.
  • Decision and Changing Steps: The number of decision-making and lane-changing steps for AugAIRL was comparable to expert values, indicating human-like maneuvering.
  • Stability: AIRL-based methods, including AugAIRL, showed greater training stability than GAIL.

Limitations and Future Work

The authors acknowledge that the semantic reward weights are trainable, providing flexibility. However, the initial assignment of semantic reward values themselves is still somewhat subjective. Future work could explore more sophisticated methods for automatically discovering or refining these semantic reward components. The paper also suggests that the learned reward function could be useful for adapting to different environments, which would be an interesting avenue for future research. Additionally, while the simulation environment is rich, real-world validation would be the ultimate test of the method's robustness.

Summary updated

Abstract

Making decisions in complex driving environments is a challenging task for autonomous agents. Imitation learning methods have great potentials for achieving such a goal. Adversarial Inverse Reinforcement Learning (AIRL) is one of the state-of-art imitation learning methods that can learn both a behavioral policy and a reward function simultaneously, yet it is only demonstrated in simple and static environments where no interactions are introduced. In this paper, we improve and stabilize AIRL's performance by augmenting it with semantic rewards in the learning framework. Additionally, we adapt the augmented AIRL to a more practical and challenging decision-making task in a highly interactive environment in autonomous driving. The proposed method is compared with four baselines and evaluated by four performance metrics. Simulation results show that the augmented AIRL outperforms all the baseline methods, and its performance is comparable with that of the experts on all of the four metrics.
BibTeX
@inproceedings{wang2021adversarial,
  author = {Pin Wang and Dapeng Liu and Jiayu Chen and Ching-Yao Chan},
  title = {{Adversarial Inverse Reinforcement Learning for Decision Making in Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560907},
}
Figure