Abstract: The technological and scientific challenges involved in the development of autonomous vehicles (AVs) are currently of primary interest for many automobile companies and research labs. However, human-controlled vehicles are likely to remain on the roads for several decades to come and may share with AVs the traffic environments of the future. In such mixed environments, AVs should deploy human-like driving policies and negotiation skills to enable smooth traffic flow. To generate automated human-like driving policies, we introduce a model-free, deep reinforcement learning approach to imitate an experienced human driver's behavior. We study a static obstacle avoidance task on a two-lane highway road in simulation (Unity). Our control algorithm receives a stochastic feedback signal from two sources: a model-driven part, encoding simple driving rules, such as lane-keeping and speed control, and a stochastic, data-driven part, incorporating human expert knowledge from driving data. To assess the similarity between machine and human driving, we model distributions of track position and speed as Gaussian processes. We demonstrate that our approach leads to human-like driving policies.
Introduction: Bridging the "Uncanny Valley" of Machine Driving
As autonomous vehicles (AVs) move from controlled testing to public roads, they must coexist with human drivers. While an AV might be mathematically optimized for safety or fuel efficiency, its behavior can often feel unpredictable or uncomfortable to humans sharing the road. This "uncanny valley" of machine driving—where a car's movements don't align with human expectations—can lead to confusion, passenger discomfort, and even accidents in mixed-traffic environments.
Humans rely on internal mental models, or "schemata," to predict the actions of others. When a machine deviates from these expected patterns (e.g., by braking abruptly for a distant obstacle or taking a path that feels unnaturally rigid), it creates friction. To address this, researchers at Ben-Gurion University of the Negev have developed a Deep Reinforcement Learning (DRL) framework designed to imitate not just the path a human takes, but the natural variability and "feel" of human driving.
Methodology: Imitating Variability with Gaussian Processes
The researchers utilized a simulation environment built in Unity, featuring a 4.7 km two-lane highway populated with 17 static obstacles. To train an agent that drives like a human, the team combined traditional driving rules with rewards derived from real human expert data.
Data Acquisition and Sensor Setup
Data was recorded from experienced drivers using a high-fidelity steering wheel and pedal setup. To navigate the environment, the DRL agent was equipped with a 310-dimensional state-space, which included 288 range-finder sensors to detect obstacles and road boundaries in its immediate vicinity.
The Hybrid Reward Function
The core innovation lies in the reward function, which consists of two parts:
- Model-driven: Penalties for objective failures like collisions, excessive slowing, or leaving the roadway.
- Data-driven (Stochastic): Instead of forcing the agent to follow a single "perfect" line, the researchers used Gaussian Processes (GPs) to model the probability distribution of a human expert's track position and speed. During training, the agent received rewards for matching samples from these distributions, allowing it to learn the acceptable "wiggle room" inherent in human driving.
To further enable this variability, the agent utilized a Mixture Density Network (MDN) as its output layer, allowing it to choose from a distribution of possible actions rather than a single deterministic "best" action.
Key Results and the Visual Turing Test
The agent successfully learned to recover the human expert's driving style, particularly in terms of track position. As shown in the results, the agent's trajectories closely mirrored the weaving patterns used by humans to navigate around obstacles.
Performance Metrics
- Track Position Accuracy: The agent demonstrated high fidelity in matching the human expert's lateral positioning ($D$), navigating the obstacle course in a fluid, human-like manner.
- Speed Recovery: While the agent followed the general rhythm of human speed changes, it tended to drive faster in certain sections. This was attributed to a "foresight gap": the agent's sensors only perceived 300 meters ahead, whereas human drivers possessed global knowledge of the track layout.
- The Visual Turing Test: To validate the "human-likeness" of the policy, the researchers conducted a blind test with robotics experts. When shown videos of the agent and a human driver, the experts were unable to distinguish between them 60% of the time, indicating that the agent's behavior was highly convincing.
Conclusion and Future Directions
This research demonstrates that DRL agents can be trained to exhibit human-like stochasticity by imitating the distribution of expert behavior rather than just the mean. While the current study focused on static obstacles, it provides a foundation for more complex scenarios involving dynamic traffic. Future work aims to incorporate Recurrent Neural Networks (RNNs) to bridge the foresight gap and use camera-based vision systems to provide a richer environmental context for the agent.
Summary updated