Randomized Adversarial Imitation Learning for Autonomous Driving

Myungjae Shin, Joongheon KimIJCAI 2019 · DOI

TL;DR

This paper introduces Randomized Adversarial Imitation Learning (RAIL), a novel approach for training autonomous driving policies that effectively coordinate multiple Advanced Driver-Assistance Systems (ADAS). RAIL leverages derivative-free optimization and adversarial imitation learning to learn robust driving behaviors from expert demonstrations, achieving performance comparable to human experts in complex highway scenarios.

What Problem Does This Paper Address?

Autonomous driving systems rely on a complex interplay of various ADAS functions like adaptive cruise control and lane keeping. Coordinating these functions effectively, especially in dynamic multi-lane highway environments, is crucial for safety and convenience. Traditional methods often struggle with the vast observation and action spaces, leading to brittle policies. While deep reinforcement learning (DRL) has shown promise, it often requires massive datasets and can produce policies that exploit loopholes in reward functions, leading to unsafe behavior. Imitation Learning (IL) offers a way to learn from expert demonstrations, but existing IL methods can be complex and data-intensive. This paper aims to bridge this gap by proposing a simpler, yet effective, IL method that can learn robust driving policies from expert data without relying on complex DRL algorithms or explicit reward function engineering.

Methods

The core of this paper is the Randomized Adversarial Imitation Learning (RAIL) algorithm. RAIL combines two key ideas:

  • Augmented Random Search (ARS): This is a derivative-free optimization method that explores the policy's parameter space by adding random noise. This approach is computationally less demanding than gradient-based methods and can lead to more robust policies.
  • Generative Adversarial Imitation Learning (GAIL): RAIL uses a discriminator network, similar to GAIL, to distinguish between expert demonstrations and the agent's generated trajectories. The discriminator effectively acts as a reward function, guiding the agent to produce behaviors that mimic the expert.

RAIL's unique contribution lies in its integration of ARS with the adversarial framework. Instead of relying on gradients, RAIL uses randomly generated noise matrices to perturb the policy parameters, creating multiple temporary policies. These policies interact with the environment, and their performance is evaluated. The discriminator then provides a reward signal based on how well these trajectories match expert demonstrations. This reward signal is used to update the original policy parameters via the ARS-like random search mechanism. The paper also highlights the use of Least Squares GAN (LS-GAN) for training the discriminator, which is shown to improve training stability compared to standard GANs.

Datasets

The paper does not introduce a new dataset. Instead, it uses expert demonstrations collected from a simulated highway driving environment. The simulation itself is built using Unity and is inspired by existing work in autonomous driving research.

Key Insights and Ideas

  • Derivative-free optimization for complex tasks: The paper demonstrates that derivative-free methods like ARS can be effectively applied to complex, high-dimensional problems like autonomous driving, challenging the notion that gradient-based methods are always superior.
  • Simplicity and robustness: RAIL offers a simpler alternative to complex DRL algorithms, making it easier to implement and potentially more reproducible. The derivative-free nature also contributes to robustness.
  • Adversarial learning without explicit rewards: By using a discriminator, RAIL learns to mimic expert behavior without needing to explicitly define a reward function for every driving scenario, which is a significant challenge in autonomous driving.
  • LS-GAN for improved stability: The adoption of LS-GAN for the discriminator is crucial for stable training, preventing the policy from getting stuck in suboptimal solutions due to noisy reward signals.

Key Results

The experiments were conducted in a simulated five-lane highway environment. The performance was evaluated based on metrics like average speed, number of overtakes, lane changes, and longitudinal/lateral rewards.

  • Competitive performance: The proposed RAIL algorithm, particularly the two-layer policy version, achieved performance comparable to expert drivers across several metrics. For instance, the stacked (two-layer) RAIL policy achieved an average speed of 70.38 km/h, close to the expert's 68.83 km/h.
  • Improved lane change behavior: RAIL policies, especially the two-layer version, demonstrated a reduction in unnecessary lane changes compared to behavioral cloning (BC), indicating more deliberate and expert-like decision-making.
  • Single-layer vs. Multi-layer: The paper found that a two-layer policy trained with RAIL performed better than a single-layer policy, suggesting that the hierarchical structure is beneficial for handling the complexity of autonomous driving observations. The single-layer policy showed about 90% of expert performance.

Limitations and Future Work

  • Simulation-based evaluation: The results are based on a simulated environment. While the simulator is inspired by real-world scenarios, further validation in real-world driving conditions is necessary.
  • Expert demonstration dependency: The performance of RAIL, like any IL method, is inherently dependent on the quality and quantity of expert demonstrations.
  • Future work: The authors suggest that their work opens up possibilities for controlling autonomous vehicles using RAIL policies, indicating that this approach has the potential to be a breakthrough in the field. Further exploration of different network architectures and more complex driving scenarios could be future directions.

Summary updated

BibTeX
@inproceedings{shin2019randomized,
  author = {Myungjae Shin and Joongheon Kim},
  title = {{Randomized Adversarial Imitation Learning for Autonomous Driving}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/638},
}
Figure