Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages

Sampo Kuutti, Richard Bowden, Saber Fallah

Abstract

The use of neural networks and reinforcement learning has become increasingly popular in autonomous vehicle control. However, the opaqueness of the resulting control policies presents a significant barrier to deploying neural network-based control in autonomous vehicles. In this paper, we present a reinforcement learning based approach to autonomous vehicle longitudinal control, where the rule-based safety cages provide enhanced safety for the vehicle as well as weak supervision to the reinforcement learning agent. By guiding the agent to meaningful states and actions, this weak supervision improves the convergence during training and enhances the safety of the final trained policy. This rule-based supervisory controller has the further advantage of being fully interpretable, thereby enabling traditional validation and verification approaches to ensure the safety of the vehicle. We compare models with and without safety cages, as well as models with optimal and constrained model parameters, and show that the weak supervision consistently improves the safety of exploration, speed of convergence, and model performance. Additionally, we show that when the model parameters are constrained or sub-optimal, the safety cages can enable a model to learn a safe driving policy even when the model could not be trained to drive through reinforcement learning alone.

Introduction

Deep Reinforcement Learning (RL) has shown great promise for autonomous vehicle control, but it faces two massive hurdles: opaqueness and safety during training. Standard RL agents learn through trial and error. In a safety-critical domain like driving, "error" means a high-speed collision. Furthermore, the resulting neural network policies are "black boxes," making it nearly impossible to guarantee they won't fail in a novel edge case. This paper addresses the need for a framework that ensures safety during the learning process while producing a final controller that is more robust and efficient than one trained through pure exploration.

Methods

The proposed approach utilizes a Deep Deterministic Policy Gradient (DDPG) algorithm for longitudinal vehicle control (acceleration and braking). The core innovation is the integration of Virtual Safety Cages, which act as interpretable, rule-based monitors.

Reinforcement Learning framework for autonomous vehicle control, featuring an Actor-Critic architecture with an LSTM layer.

Virtual Safety Cages

Two safety cages monitor the environment using Time Headway (TH) and Time-To-Collision (TTC). If the RL agent issues a command that breaches safety thresholds (e.g., getting too close to the car in front), the cage overrides the action with a conservative braking maneuver. These cages provide weak supervision: when a cage intervenes, the agent receives a small negative reward penalty ($r_{sc} = -0.1$), guiding it away from unsafe action spaces early in training.

Architecture and Acceleration

The agent's actor network includes an LSTM (Long Short-Term Memory) layer to capture temporal correlations in driving behavior. To accelerate training, the authors employed a neural network "World Model" trained on 45 hours of data from the IPG CarMaker simulator, allowing for high-speed GPU-based simulation of vehicle dynamics.

Datasets and Evaluation

The study relies on high-fidelity simulations and diverse testing scenarios:

  • IPG CarMaker: Used to generate over 2.3 million time-steps of driving data for World Model training and policy evaluation.
  • Naturalistic Driving: 120 test scenarios (10 hours total) mimicking everyday highway behavior.
  • Adversarial Testing Framework (ATF): A specialized environment where an adversarial RL agent attempts to find weaknesses in the controller to force a collision.

Key Insights and Results

The integration of safety cages yielded several critical improvements:

  • Pruning the Action Space: By preventing disastrous actions, safety cages allow the agent to focus on refining smooth, efficient driving rather than repeatedly learning from collisions.
  • Zero-Collision Training: The DDPG model with safety cages experienced zero collisions during the 5000-episode training process, compared to 30 collisions for the standard model.
  • Faster Convergence: Models with safety cages reached optimal reward levels in approximately 7,500 episodes, significantly faster than standard models.
  • The "Shallow Model" Rescue: Safety cages enabled simple 1-layer networks (without LSTMs) to learn safe policies. Without cages, these "shallow" models failed to learn and crashed repeatedly.
  • Safety Persistence: Even when safety cages were disabled during testing, models trained with them maintained higher minimum headways (1.69s vs. 1.53s), indicating that safety was successfully encoded into the network weights.

Conclusion

This work demonstrates that rule-based "Virtual Safety Cages" can effectively bridge the gap between the flexibility of RL and the safety requirements of autonomous driving. While currently focused on longitudinal control, future work aims to extend this framework to lateral control (steering) and more complex urban environments involving pedestrians and intersections.

Summary updated

Abstract

The use of neural networks and reinforcement learning has become increasingly popular in autonomous vehicle control. However, the opaqueness of the resulting control policies presents a significant barrier to deploying neural network-based control in autonomous vehicles. In this paper, we present a reinforcement learning based approach to autonomous vehicle longitudinal control, where the rule-based safety cages provide enhanced safety for the vehicle as well as weak supervision to the reinforcement learning agent. By guiding the agent to meaningful states and actions, this weak supervision improves the convergence during training and enhances the safety of the final trained policy. This rule-based supervisory controller has the further advantage of being fully interpretable, thereby enabling traditional validation and verification approaches to ensure the safety of the vehicle. We compare models with and without safety cages, as well as models with optimal and constrained model parameters, and show that the weak supervision consistently improves the safety of exploration, speed of convergence, and model performance. Additionally, we show that when the model parameters are constrained or sub-optimal, the safety cages can enable a model to learn a safe driving policy even when the model could not be trained to drive through reinforcement learning alone.
BibTeX
@inproceedings{kuutti2021weakly,
  author = {Sampo Kuutti and Richard Bowden and Saber Fallah},
  title = {{Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=RpBT7w8eiu},
}
Figure