From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learning

Li Zeqiao, Yijing Wang, Haoyu Wang, Zheng Li, Peng Li, Wenfei Liu, Zhiqiang ZuoarXiv 2025 · DOI · arXiv

Abstract

Autonomous driving with reinforcement learning (RL) has significant potential. However, applying RL in real-world settings remains challenging due to the need for safe, efficient, and robust learning. Incorporating human expertise into the learning process can help overcome these challenges by reducing risky exploration and improving sample efficiency. In this work, we propose a reward-free, active human-in-the-loop learning method called Human-Guided Distributional Soft Actor-Critic (H-DSAC). Our method combines Proxy Value Propagation (PVP) and Distributional Soft Actor-Critic (DSAC) to enable efficient and safe training in real-world environments. The key innovation is the construction of a distributed proxy value function within the DSAC framework. This function encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention. By extrapolating these labels to unlabeled states, the policy is effectively guided toward expert-like behavior. With a well-designed state space, our method achieves real-world driving policy learning within practical training times. Results from both simulation and real-world experiments demonstrate that our framework enables safe, robust, and sample-efficient learning for autonomous driving.

Introduction

Reinforcement Learning (RL) offers a promising path toward teaching autonomous agents through trial and error. However, its application to real-world autonomous driving (AD) is hindered by three critical challenges: safety, sample efficiency, and reward design. In a physical environment, "error" can lead to catastrophic collisions. Furthermore, standard RL often requires millions of interactions, which is impractical for real-time vehicle training. Finally, defining a mathematical reward function that captures the nuances of "good driving" without inducing "reward hacking" is notoriously difficult.

To address these issues, we introduce H-DSAC (Human-Guided Distributional Soft Actor-Critic), a framework that leverages human interventions as a training signal to achieve safe and efficient learning directly in the real world.

The H-DSAC Framework

The core of H-DSAC is the integration of Distributional Soft Actor-Critic (DSAC) with Proxy Value Propagation (PVP). This combination allows the agent to learn from human behavior without requiring a manually tuned reward function.

Overview of the H-DSAC framework, illustrating the interaction between the human expert, the environment, and the agent's policy learning components.

Distributed Proxy Value Function

The system's primary innovation is the Distributed Proxy Value Function. Instead of traditional rewards, the agent learns from human-labeled intent:

  • Expert Demonstrations: Actions provided by a human are labeled with a value of +1.
  • Human Interventions: When a human intervenes to prevent a mistake, the agent's intended action is labeled -1.

Unlike standard RL, which predicts an average return, H-DSAC models the entire probability distribution of possible outcomes. This distributional approach enables the agent to quantify uncertainty and risk, which is vital for safety-critical maneuvers. Through Temporal-Difference (TD) learning, these human-provided labels are "propagated" to unlabeled states, allowing the agent to infer human-aligned behavior even in novel scenarios.

Detailed view of the Policy Learning mechanism, showing the use of human and novice buffers to train the distributional proxy value function.

Experimental Evaluation

The H-DSAC framework was evaluated in both simulation and real-world settings:

  1. MetaDrive Safety Benchmark: A high-fidelity simulator used to test generalization across 30 scenarios (20 training, 10 testing) featuring various obstacles and road blocks.
  2. Real-World UGV Platform: An Unmanned Ground Vehicle equipped with LiDAR, Radar, and cameras was trained on the campus roads of Tianjin University.

Key Results

  • Superior Performance and Safety: In the MetaDrive simulator, H-DSAC achieved an 83% success rate, outperforming standard SAC (73%). Crucially, it recorded the lowest episodic safety cost (0.31) compared to standard RL methods like PPO (3.92), indicating significantly fewer crashes during training.
  • Real-World Efficiency: The UGV mastered autonomous navigation on campus roads in approximately 2 hours (100,000 steps).
  • Robust Generalization: After training on a primary route, the vehicle successfully navigated a more complex second route featuring pedestrians, cyclists, and heavy traffic without additional training.

Conclusion and Future Work

H-DSAC demonstrates that treating human interventions as data—rather than just safety nets—can dramatically accelerate RL training for autonomous driving. By modeling the variance of returns, the agent becomes more robust to real-world uncertainties.

While the current system significantly reduces human burden, it still requires a supervisor during the training phase. Future research will focus on further reducing supervision time and transitioning from LiDAR-based state representations to end-to-end vision-based learning (pixels-to-control) to handle even more complex urban environments.

Summary updated

Abstract

Autonomous driving with reinforcement learning (RL) has significant potential. However, applying RL in real-world settings remains challenging due to the need for safe, efficient, and robust learning. Incorporating human expertise into the learning process can help overcome these challenges by reducing risky exploration and improving sample efficiency. In this work, we propose a reward-free, active human-in-the-loop learning method called Human-Guided Distributional Soft Actor-Critic (H-DSAC). Our method combines Proxy Value Propagation (PVP) and Distributional Soft Actor-Critic (DSAC) to enable efficient and safe training in real-world environments. The key innovation is the construction of a distributed proxy value function within the DSAC framework. This function encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention. By extrapolating these labels to unlabeled states, the policy is effectively guided toward expert-like behavior. With a well-designed state space, our method achieves real-world driving policy learning within practical training times. Results from both simulation and real-world experiments demonstrate that our framework enables safe, robust, and sample-efficient learning for autonomous driving.
BibTeX
@misc{zeqiao2025from,
  author = {Li Zeqiao and Yijing Wang and Haoyu Wang and Zheng Li and Peng Li and Wenfei Liu and Zhiqiang Zuo},
  title = {{From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learning}},
  year = {2025},
  eprint = {2510.06038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.06038},
  howpublished = {arXiv},
}
Figure