Abstract
Autonomous driving with reinforcement learning (RL) has significant potential. However, applying RL in real-world settings remains challenging due to the need for safe, efficient, and robust learning. Incorporating human expertise into the learning process can help overcome these challenges by reducing risky exploration and improving sample efficiency. In this work, we propose a reward-free, active human-in-the-loop learning method called Human-Guided Distributional Soft Actor-Critic (H-DSAC). Our method combines Proxy Value Propagation (PVP) and Distributional Soft Actor-Critic (DSAC) to enable efficient and safe training in real-world environments. The key innovation is the construction of a distributed proxy value function within the DSAC framework. This function encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention. By extrapolating these labels to unlabeled states, the policy is effectively guided toward expert-like behavior. With a well-designed state space, our method achieves real-world driving policy learning within practical training times. Results from both simulation and real-world experiments demonstrate that our framework enables safe, robust, and sample-efficient learning for autonomous driving.
Introduction
Reinforcement Learning (RL) offers a promising path toward teaching autonomous agents through trial and error. However, its application to real-world autonomous driving (AD) is hindered by three critical challenges: safety, sample efficiency, and reward design. In a physical environment, "error" can lead to catastrophic collisions. Furthermore, standard RL often requires millions of interactions, which is impractical for real-time vehicle training. Finally, defining a mathematical reward function that captures the nuances of "good driving" without inducing "reward hacking" is notoriously difficult.
To address these issues, we introduce H-DSAC (Human-Guided Distributional Soft Actor-Critic), a framework that leverages human interventions as a training signal to achieve safe and efficient learning directly in the real world.
The H-DSAC Framework
The core of H-DSAC is the integration of Distributional Soft Actor-Critic (DSAC) with Proxy Value Propagation (PVP). This combination allows the agent to learn from human behavior without requiring a manually tuned reward function.
Distributed Proxy Value Function
The system's primary innovation is the Distributed Proxy Value Function. Instead of traditional rewards, the agent learns from human-labeled intent:
- Expert Demonstrations: Actions provided by a human are labeled with a value of +1.
- Human Interventions: When a human intervenes to prevent a mistake, the agent's intended action is labeled -1.
Unlike standard RL, which predicts an average return, H-DSAC models the entire probability distribution of possible outcomes. This distributional approach enables the agent to quantify uncertainty and risk, which is vital for safety-critical maneuvers. Through Temporal-Difference (TD) learning, these human-provided labels are "propagated" to unlabeled states, allowing the agent to infer human-aligned behavior even in novel scenarios.
Experimental Evaluation
The H-DSAC framework was evaluated in both simulation and real-world settings:
- MetaDrive Safety Benchmark: A high-fidelity simulator used to test generalization across 30 scenarios (20 training, 10 testing) featuring various obstacles and road blocks.
- Real-World UGV Platform: An Unmanned Ground Vehicle equipped with LiDAR, Radar, and cameras was trained on the campus roads of Tianjin University.
Key Results
- Superior Performance and Safety: In the MetaDrive simulator, H-DSAC achieved an 83% success rate, outperforming standard SAC (73%). Crucially, it recorded the lowest episodic safety cost (0.31) compared to standard RL methods like PPO (3.92), indicating significantly fewer crashes during training.
- Real-World Efficiency: The UGV mastered autonomous navigation on campus roads in approximately 2 hours (100,000 steps).
- Robust Generalization: After training on a primary route, the vehicle successfully navigated a more complex second route featuring pedestrians, cyclists, and heavy traffic without additional training.
Conclusion and Future Work
H-DSAC demonstrates that treating human interventions as data—rather than just safety nets—can dramatically accelerate RL training for autonomous driving. By modeling the variance of returns, the agent becomes more robust to real-world uncertainties.
While the current system significantly reduces human burden, it still requires a supervisor during the training phase. Future research will focus on further reducing supervision time and transitioning from LiDAR-based state representations to end-to-end vision-based learning (pixels-to-control) to handle even more complex urban environments.
Summary updated