Abstract
Making decisions in complex driving environments is a challenging task for autonomous agents. Imitation learning methods have great potentials for achieving such a goal. Adversarial Inverse Reinforcement Learning (AIRL) is one of the state-of-art imitation learning methods that can learn both a behavioral policy and a reward function simultaneously, yet it is only demonstrated in simple and static environments where no interactions are introduced. In this paper, we improve and stabilize AIRL's performance by augmenting it with semantic rewards in the learning framework. Additionally, we adapt the augmented AIRL to a more practical and challenging decision-making task in a highly interactive environment in autonomous driving. The proposed method is compared with four baselines and evaluated by four performance metrics. Simulation results show that the augmented AIRL outperforms all the baseline methods, and its performance is comparable with that of the experts on all of the four metrics.
Introduction
Reinforcement Learning (RL) has shown immense promise in robotics, yet it often suffers from the "reward engineering" bottleneck. Defining a mathematical reward function that perfectly balances competing objectives like safety, speed, and comfort is notoriously difficult. Inverse Reinforcement Learning (IRL) addresses this by inferring the reward function from expert demonstrations. However, state-of-the-art methods like Adversarial Inverse Reinforcement Learning (AIRL) frequently struggle in highly interactive, dynamic environments such as highway traffic, where other agents react to the learner's actions.
To bridge this gap, researchers have developed Augmented Adversarial Inverse Reinforcement Learning (AugAIRL). By incorporating "semantic rewards"—high-level, sparse signals representing fundamental driving principles—into the adversarial learning process, the framework stabilizes training and enables agents to master complex lane-changing maneuvers that match human-like performance.
Methodology: Augmented AIRL
The core innovation of AugAIRL lies in its dual-pronged approach to stabilizing imitation learning in interactive settings.
Semantic Reward Augmentation
Standard AIRL relies on a discriminator to distinguish between expert and generated trajectories. In AugAIRL, the discriminator is "nudged" with semantic rewards ($r_{sem}$). These are sparse, high-level signals such as penalties for collisions (-30) or safety margin violations (-1), and bonuses for successful task completion (+15) or lateral movement (+0.3).
Crucially, these rewards are integrated into the discriminator's helper function. This reparameterization synchronizes the learning of the reward function and the behavioral policy more effectively, allowing the model to capture the true underlying intent of the expert.
Hierarchical Decision-Making
To handle the complexity of highway driving, the model employs a hierarchical structure. Instead of learning raw control inputs like steering or acceleration, the agent operates in a high-level action space: selecting the optimal "gap" in traffic to target. Once a gap is chosen, low-level controllers execute the physical maneuvers. This abstraction makes the AI's decisions more explainable and reduces the noise in the learning process.
Experimental Setup
The proposed method was evaluated in a custom highway simulation featuring three lanes and dynamic traffic. The state space consisted of 44 features, including the kinematics of the ego vehicle and five surrounding vehicles. Expert data was generated using a rule-based system with full state access. The simulation randomized traffic density and the "personalities" of surrounding drivers to ensure robustness.
Results and Discussion
AugAIRL (denoted as AIRL+SPARSE in results) was compared against standard AIRL, GAIL (Generative Adversarial Imitation Learning), and other baselines.
As shown in the training curves, AugAIRL significantly outperforms the baselines:
- Success Ratio: AugAIRL achieved a perfect 1.0 (100%) success rate, surpassing standard AIRL (0.99) and GAIL (0.95).
- Total Reward: The mean reward reached 22.12, closely approaching the expert's performance of 24.21.
- Stability: AugAIRL exhibited much lower variance during training compared to standard AIRL, indicating a more stable convergence.
- Human-like Behavior: The agent learned sophisticated behaviors, such as the ability to abort a lane change if risks increased—a hallmark of human-like decision-making often absent in simpler models.
Conclusion
By augmenting the AIRL framework with semantic rewards, this research demonstrates a more stable and effective method for imitation learning in interactive environments. While currently validated in simulation, future work will focus on addressing the sim-to-real gap and scaling the approach to more complex urban scenarios. The use of semantic "hints" provides a promising path toward explainable and reliable autonomous driving systems.
Summary updated