Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

Quanyi Li, Zhenghao Peng, Bolei Zhou

Abstract

Human intervention is an effective way to inject human knowledge into the training loop of reinforcement learning, which can bring fast learning and ensured training safety. Given the very limited budget of human intervention, it remains challenging to design when and how human expert interacts with the learning agent in the training. In this work, we develop a novel human-in-the-loop learning method called Human-AI Copilot Optimization (HACO). To allow the agent's sufficient exploration in the risky environments while ensuring the training safety, the human expert can take over the control and demonstrate how to avoid probably dangerous situations or trivial behaviors. The proposed HACO then effectively utilizes the data both from the trial-and-error exploration and human's partial demonstration to train a high-performing agent. HACO extracts proxy state-action values from partial human demonstration and optimizes the agent to improve the proxy values meanwhile reduce the human interventions. The experiments show that HACO achieves a substantially high sample efficiency in the safe driving benchmark. HACO can train agents to drive in unseen traffic scenarios with a handful of human intervention budget and achieve high safety and generalizability, outperforming both reinforcement learning and imitation learning baselines with a large margin.

Introduction

Training autonomous driving agents usually requires millions of trials (and many crashes) or massive datasets of perfect human driving. Reinforcement Learning (RL) and Imitation Learning (IL) are the two pillars of autonomous driving research, but both have significant flaws for real-world deployment. RL relies on dangerous trial-and-error, making it difficult to define reward functions that capture driving nuances without causing frequent collisions. Conversely, IL is brittle due to "distributional shift," where agents fail to recover from states not present in the training data.

Human-AI Copilot Optimization (HACO) introduces a "copilot" paradigm where a human expert proactively intervenes only when necessary. This allows the agent to learn safe, high-performing policies in just 30,000 steps—a 30x improvement in efficiency over standard reinforcement learning—without requiring any environmental reward functions.

The HACO Framework

HACO treats the human and AI as a collaborative team. The agent explores the environment freely, but if a human expert observes the agent approaching a mistake (e.g., heading toward a curb), they take over, demonstrate the correct recovery, and hand control back.

The HACO framework: (A) System architecture, (B) Physical human interface, and (C) Visualization of the copilot trajectory and intervention points.

Key technical components include:

  • Proxy Value Function: Instead of a manual reward function, HACO uses offline RL to learn a "proxy" value function $Q(s, a)$ that is optimistic toward human actions and pessimistic toward rejected agent actions.
  • Intervention Minimization: To ensure autonomy, human intervention is treated as a cost. The agent is optimized to maximize proxy value while minimizing this cost.
  • Cosine Similarity Cost: HACO uses the cosine similarity between the agent's intended action and the human's corrective action to provide a smooth gradient for learning.

Experimental Evaluation

The researchers evaluated HACO using two major driving simulators:

  1. MetaDrive: A lightweight simulator using procedural generation to create infinite traffic scenes. Evaluation involved 50 training scenes and 50 unseen testing scenes.
  2. CARLA: A high-fidelity urban simulator where agents were trained in "Town 1" and tested in "Town 2" to validate performance with complex visual inputs.
MetaDrive scenarios used for training and testing, including roundabouts, highways, and complex intersections.

Key Results and Insights

  • Sample Efficiency: HACO reached a high success rate in just 30,000 steps (approx. 50 minutes of human time), compared to 1,000,000 steps for standard RL.
  • Safety: During training, HACO recorded only ~30 safety violations, whereas standard RL agents crashed over 2,700 times.
  • Generalization: In unseen test maps, HACO achieved an 83% success rate, significantly outperforming IL and Safe RL baselines.
  • The "Human Abuse" Problem: HACO prevents agents from deliberately driving into danger to force human takeovers by explicitly minimizing intervention costs.
  • Trajectory Learning: Unlike step-wise Behavior Cloning, HACO incentivizes actions leading to high-value future states, increasing robustness.

Conclusion

HACO demonstrates that human knowledge can be injected efficiently while keeping the human "cognitive budget" low and the environment safe. While the resulting agents can be conservative—mimicking the cautious rescue maneuvers of human experts—the method provides a viable path for training on actual hardware. Future work will focus on teaching more assertive skills and further reducing human load through learned "AI guardians."

Summary updated

Abstract

Human intervention is an effective way to inject human knowledge into the training loop of reinforcement learning, which can bring fast learning and ensured training safety. Given the very limited budget of human intervention, it remains challenging to design when and how human expert interacts with the learning agent in the training. In this work, we develop a novel human-in-the-loop learning method called Human-AI Copilot Optimization (HACO).To allow the agent's sufficient exploration in the risky environments while ensuring the training safety, the human expert can take over the control and demonstrate how to avoid probably dangerous situations or trivial behaviors. The proposed HACO then effectively utilizes the data both from the trial-and-error exploration and human's partial demonstration to train a high-performing agent. HACO extracts proxy state-action values from partial human demonstration and optimizes the agent to improve the proxy values meanwhile reduce the human interventions. The experiments show that HACO achieves a substantially high sample efficiency in the safe driving benchmark. HACO can train agents to drive in unseen traffic scenarios with a handful of human intervention budget and achieve high safety and generalizability, outperforming both reinforcement learning and imitation learning baselines with a large margin. Code and demo videos are available at: https://decisionforce.github.io/HACO/.
BibTeX
@inproceedings{li2022efficient,
  author = {Quanyi Li and Zhenghao Peng and Bolei Zhou},
  title = {{Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=0cgU-BZp2ky},
}
Figure