Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Zhiyu Huang, Jingda Wu, Chen Lv

Abstract: Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.

Introduction

Reinforcement learning (RL) has emerged as a powerful tool for complex decision-making in autonomous driving. However, two significant bottlenecks persist: low sample efficiency and the difficulty of reward function design. Standard RL agents often require millions of interactions to learn, and defining "good driving" mathematically frequently leads to "reward hacking," where agents exploit loopholes to maximize rewards without achieving safe or natural behavior. This research addresses these challenges by continuously injecting human prior knowledge into the RL training loop to guide exploration and eliminate the need for complex reward engineering.

Methodology

The proposed framework integrates human expertise into the learning process through a structured three-step approach.

The proposed three-step framework: Expert demonstration, Policy derivation, and Reinforcement learning with imitative priors.
  1. Expert Demonstration: A human expert performs simulated driving tasks, recording state-action pairs ($s_t, a_t^E$).
  2. Policy Derivation: Rather than simple deterministic cloning, the system trains a stochastic expert policy using a Deep Ensemble. This captures both Policy Uncertainty (inherent randomness in human choice) and Model Uncertainty (lack of confidence in unfamiliar states).
  3. Reinforcement Learning with Expert Priors: The agent, based on Soft Actor-Critic (SAC), is regularized to stay near the expert policy using Kullback–Leibler (KL) divergence. This is implemented via either a Value Penalty (modifying the reward) or a Policy Constraint (using Lagrangian dual gradient descent to bound policy updates).

Network Architecture

The agent processes environmental data through a sophisticated neural network designed for spatial and temporal awareness. The state input consists of an 80x80x9 Bird's-Eye View (BEV) image stack, representing the current and two previous timesteps.

Neural network architecture featuring a CNN state encoder and separate heads for Q-value, Policy, and Value networks.

As shown in the architecture diagram, a CNN state encoder extracts features which are then branched into three distinct heads: a Q-value network (incorporating action input), a policy network (outputting mean $\mu$ and standard deviation $\sigma$), and a value network.

Experimental Setup and Results

The framework was evaluated using the SMARTS simulation platform across two challenging urban scenarios: an unprotected left turn and a multi-lane roundabout. For the left turn, two distinct datasets were used to test style transfer: Aggressive (nudging into traffic) and Conservative (waiting for gaps).

Key Findings

  • Efficiency Gains: The method achieved standard SAC performance using only ~30% of the interaction steps, representing a 70% improvement in sample efficiency.
  • Performance: The "Value Penalty" variant reached success rates of 96% in left-turn scenarios and 84% in roundabouts, significantly outperforming baselines like PPO and GAIL which struggled with sparse rewards.
  • Sparse Reward Sufficiency: By using expert priors to restrict nonsensical actions, the agent successfully learned using only simple, sparse rewards (+1 for goal, -1 for collision), bypassing complex reward engineering.
  • Human-like Behavior: The framework successfully transferred specific driving styles. Aggressive-trained agents learned to force yields, while conservative-trained agents waited patiently, both exhibiting smoother control profiles than pure RL agents.

Conclusion

By leveraging uncertainty-aware imitative priors, this framework significantly reduces the data requirements for training autonomous driving agents while ensuring human-like behavior. Future work will focus on adapting the framework for naturalistic raw trajectory data and generalizing to a broader range of complex real-world scenarios.

Summary updated

Abstract

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.
BibTeX
@article{huang2023efficient,
  author = {Zhiyu Huang and Jingda Wu and Chen Lv},
  title = {{Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=G9b2W1PzTCs},
}
Figure