Quantitative and Qualitative Evaluation of Reinforcement Learning Policies for Autonomous Vehicles

Laura Ferrarotti, Massimiliano Luca, Gabriele Santin, Giorgio Previati, Gianpiero Mastinu, Elena Campi, Lorenzo Uccello, Antonino Albanese, Praveen Zalaya, Alessandro Roccasalva, Bruno Lepri

Abstract

This study evaluates Reinforcement Learning (RL) policies for autonomous vehicles (AVs) navigating a real-world roundabout in Milan, Italy. Utilizing a "human-in-the-loop" framework, the research combines high-fidelity traffic and driving simulators with a physical cockpit to assess both quantitative traffic metrics and qualitative human perception. Results indicate that increasing AV penetration significantly reduces travel time, fuel consumption, and emissions. Crucially, human drivers reported feeling safer and perceiving traffic as smoother in high-AV scenarios, suggesting that optimized RL policies create positive "ripple effects" for all road users.

Introduction

The deployment of autonomous vehicles (AVs) in mixed-traffic environments presents significant challenges, particularly in complex urban layouts like roundabouts. While Reinforcement Learning (RL) has proven effective for training AV policies in isolated simulations, these "vacuum" environments often fail to account for the unpredictable behavior of human-driven vehicles (HVs) and the subjective comfort of human passengers.

A critical concern in mixed autonomy is the "sim-to-real" gap: will a mathematically optimal policy for an AV be perceived as erratic or dangerous by a human driver in an adjacent lane? This research addresses these questions by conducting a dual-mode evaluation—quantitative analysis of traffic flow and qualitative study of human perception—using a professional-grade driving simulator at the Polytechnic University of Milan.

Methods

The researchers developed a multi-layered co-simulation framework to bridge the gap between AI training and human experience.

Simulation Framework

  1. Traffic Modeling: SUMO (Simulation of Urban MObility) was used for microscopic traffic modeling, handling the physics and logic of multiple interacting agents.
  2. Visual Environment: VI-WorldSim provided a high-fidelity graphical environment for the driving experience.
  3. Physical Hardware: The DrisMi Lab at the Polytechnic University of Milan provided a cable-driven cockpit with realistic motion feedback, allowing human participants to interact with the AI-controlled agents in real-time.
The simulated environment of the Milan roundabout used for testing RL policies.

RL Policy and Calibration

The AVs were trained using Proximal Policy Optimization (PPO) with a multi-objective reward function designed to minimize crossing time and polluting emissions. To ensure realism, the simulation was calibrated using field measurements from a specific roundabout in Milan, including queue lengths and flow rates for cars, pedestrians, and bicycles. A key technical achievement was the synchronized co-simulation between SUMO and VI-WorldSim, maintaining a latency of just 5ms to prevent "simulator sickness" or lag for the human driver.

The high-fidelity, cable-driven cockpit used for human-in-the-loop qualitative evaluation at the DrisMi Lab.

Results and Discussion

The study revealed that AVs do not merely optimize their own performance; they act as "traffic pacers" that improve the environment for human drivers.

Quantitative Performance

As AV penetration increased, significant improvements were observed across all metrics:

  • Efficiency: At 100% AV penetration, crossing times for AVs were reduced by 10.72%. Interestingly, HVs also saw an 8.52% reduction in travel time, benefiting from the smoother flow established by the AI.
  • Sustainability: The RL policy's focus on smooth acceleration led to a 39% reduction in emissions and a 35% drop in fuel consumption at full penetration. Even at 50% penetration, total emissions decreased by approximately 25-30%.

Qualitative Human Perception

The "human-in-the-loop" study provided unique insights into how people perceive mixed autonomy:

  • Safety and Predictability: 80% of participants felt that scenarios with high AV penetration (80%) were safer than those with low penetration (20%).
  • Preference: 70% of participants preferred driving in the high-AV environment, with 50% specifically noting that the traffic felt "smoother." This suggests that RL policies optimized for efficiency also tend to produce more predictable and comfortable behaviors for human observers.

Conclusion

This research demonstrates that RL-trained AVs can significantly enhance urban traffic efficiency and sustainability while maintaining—and even improving—human comfort and perceived safety. The findings highlight the importance of "altruistic" AI behavior in mixed-traffic scenarios. Future work will focus on expanding the participant pool for qualitative studies and testing the robustness of these policies across diverse urban layouts and adverse weather conditions.

Summary updated

BibTeX
@article{ferrarotti2024quantitative,
  author = {Laura Ferrarotti and Massimiliano Luca and Gabriele Santin and Giorgio Previati and Gianpiero Mastinu and Elena Campi and Lorenzo Uccello and Antonino Albanese and Praveen Zalaya and Alessandro Roccasalva and Bruno Lepri},
  title = {{Quantitative and Qualitative Evaluation of Reinforcement Learning Policies for Autonomous Vehicles}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3370469},
}
Figure