Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning

Pin Wang, Dapeng Liu, Jiayu Chen, Hanhan Li, Ching‐Yao Chan

Abstract

Making decisions in complex driving environments is a challenging task for autonomous agents. Imitation learning methods have great potentials for achieving such a goal. Adversarial Inverse Reinforcement Learning (AIRL) is one of the state-of-art imitation learning methods that can learn both a behavioral policy and a reward function simultaneously, yet it is only demonstrated in simple and static environments where no interactions are introduced. In this paper, we improve and stabilize AIRL's performance by augmenting it with semantic rewards in the learning framework. Additionally, we adapt the augmented AIRL to a more practical and challenging decision-making task in a highly interactive environment in autonomous driving. The proposed method is compared with four baselines and evaluated by four performance metrics. Simulation results show that the augmented AIRL outperforms all the baseline methods, and its performance is comparable with that of the experts on all of the four metrics.

Introduction

Reinforcement Learning (RL) has shown immense promise in robotics, yet it often suffers from the "reward engineering" bottleneck. Defining a mathematical reward function that perfectly balances competing objectives like safety, speed, and comfort is notoriously difficult. Inverse Reinforcement Learning (IRL) addresses this by inferring the reward function from expert demonstrations. However, state-of-the-art methods like Adversarial Inverse Reinforcement Learning (AIRL) frequently struggle in highly interactive, dynamic environments such as highway traffic, where other agents react to the learner's actions.

To bridge this gap, researchers have developed Augmented Adversarial Inverse Reinforcement Learning (AugAIRL). By incorporating "semantic rewards"—high-level, sparse signals representing fundamental driving principles—into the adversarial learning process, the framework stabilizes training and enables agents to master complex lane-changing maneuvers that match human-like performance.

Methodology: Augmented AIRL

The core innovation of AugAIRL lies in its dual-pronged approach to stabilizing imitation learning in interactive settings.

Semantic Reward Augmentation

Standard AIRL relies on a discriminator to distinguish between expert and generated trajectories. In AugAIRL, the discriminator is "nudged" with semantic rewards ($r_{sem}$). These are sparse, high-level signals such as penalties for collisions (-30) or safety margin violations (-1), and bonuses for successful task completion (+15) or lateral movement (+0.3).

Crucially, these rewards are integrated into the discriminator's helper function. This reparameterization synchronizes the learning of the reward function and the behavioral policy more effectively, allowing the model to capture the true underlying intent of the expert.

The AugAIRL framework integrates semantic terms into the discriminator to guide the learning of the reward function and policy.

Hierarchical Decision-Making

To handle the complexity of highway driving, the model employs a hierarchical structure. Instead of learning raw control inputs like steering or acceleration, the agent operates in a high-level action space: selecting the optimal "gap" in traffic to target. Once a gap is chosen, low-level controllers execute the physical maneuvers. This abstraction makes the AI's decisions more explainable and reduces the noise in the learning process.

Experimental Setup

The proposed method was evaluated in a custom highway simulation featuring three lanes and dynamic traffic. The state space consisted of 44 features, including the kinematics of the ego vehicle and five surrounding vehicles. Expert data was generated using a rule-based system with full state access. The simulation randomized traffic density and the "personalities" of surrounding drivers to ensure robustness.

Results and Discussion

AugAIRL (denoted as AIRL+SPARSE in results) was compared against standard AIRL, GAIL (Generative Adversarial Imitation Learning), and other baselines.

Performance comparison of AugAIRL (AIRL+SPARSE) against AIRL and GAIL across four metrics: total reward, success ratio, decision steps, and changing steps.

As shown in the training curves, AugAIRL significantly outperforms the baselines:

  • Success Ratio: AugAIRL achieved a perfect 1.0 (100%) success rate, surpassing standard AIRL (0.99) and GAIL (0.95).
  • Total Reward: The mean reward reached 22.12, closely approaching the expert's performance of 24.21.
  • Stability: AugAIRL exhibited much lower variance during training compared to standard AIRL, indicating a more stable convergence.
  • Human-like Behavior: The agent learned sophisticated behaviors, such as the ability to abort a lane change if risks increased—a hallmark of human-like decision-making often absent in simpler models.

Conclusion

By augmenting the AIRL framework with semantic rewards, this research demonstrates a more stable and effective method for imitation learning in interactive environments. While currently validated in simulation, future work will focus on addressing the sim-to-real gap and scaling the approach to more complex urban scenarios. The use of semantic "hints" provides a promising path toward explainable and reliable autonomous driving systems.

Summary updated

Abstract

Making decisions in complex driving environments is a challenging task for autonomous agents. Imitation learning methods have great potentials for achieving such a goal. Adversarial Inverse Reinforcement Learning (AIRL) is one of the state-of-art imitation learning methods that can learn both a behavioral policy and a reward function simultaneously, yet it is only demonstrated in simple and static environments where no interactions are introduced. In this paper, we improve and stabilize AIRL's performance by augmenting it with semantic rewards in the learning framework. Additionally, we adapt the augmented AIRL to a more practical and challenging decision-making task in a highly interactive environment in autonomous driving. The proposed method is compared with four baselines and evaluated by four performance metrics. Simulation results show that the augmented AIRL outperforms all the baseline methods, and its performance is comparable with that of the experts on all of the four metrics.
BibTeX
@inproceedings{wang2021decision,
  author = {Pin Wang and Dapeng Liu and Jiayu Chen and Hanhan Li and Ching‐Yao Chan},
  title = {{Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560907},
}
Figure