LEE: A Photorealistic Virtual Environment for Assessing Driver-Vehicle Interactions in Self-driving Mode

Saad Minhas, Aura Hernández-Sabaté, Shoaib Ehsan, Katerine Díaz-Chito, Ales Leonardis, Antonio M. López, Klaus D. McDonald-MaierWorkshops @ ECCV 2016 · DOI

Introduction

As SAE Level 3 autonomous driving systems become more prevalent, the transition of control from vehicle to human driver presents a significant safety challenge. Level 3 systems handle dynamic driving tasks but require human intervention when encountering unresolved situations. Testing these transitions safely and realistically often requires industrial-grade simulators costing tens of millions of dollars, creating a high barrier for research.

This paper introduces LEE, a low-cost, photorealistic virtual environment designed to study Perception Reaction Time (PRT) during sudden autonomous takeovers. Built with off-the-shelf gaming hardware and the Unity3D engine, LEE provides an accessible platform for high-fidelity driver-vehicle interaction research.

The LEE Simulator

The authors developed the LEE simulator by integrating accessible hardware with professional game-development tools to balance cost and realism.

Hardware and Software Architecture

The system runs on a high-end PC equipped with an Intel i7 processor and an NVIDIA GTX Titan GPU. The physical interface includes dual HD monitors, a Logitech G27 racing wheel and pedal set, and a webcam for driver monitoring.

The LEE simulator setup, showing the physical hardware (a) and the software interface (b) including real-time technical data and driver monitoring.

On the software side, the simulation is powered by Unity3D. Custom 3D assets, including a detailed Mazda 3 interior and a 4-mile looping highway modeled after London's M25, were created using Autodesk 3DS Max and Photoshop. To simulate realistic road conditions, an Intelligent Traffic System (ITS) plugin manages surrounding vehicles with randomized behaviors.

Overview of the virtual environment assets and the 4-mile looping highway track layout used for the simulation.

Experimental Methodology

To validate the utility of the LEE environment, an experiment was conducted with 10 subjects (ages 26–62). Each participant completed 12 sessions across two primary scenarios:

  1. Baseline: The driver was attentive with hands on the wheel but not focused on the road.
  2. Distracted: The driver was actively engaged with a smartphone.

During autonomous operation, a random roadblock would appear at 60m or 80m, triggering visual and acoustic alarms. The system then recorded the driver's reaction times for both hands (steering) and feet (braking).

Key Results and Insights

The study yielded several critical findings regarding human performance during emergency takeovers:

  • Asymmetric Reaction Times: Drivers instinctively prioritized braking over steering. In the baseline scenario, the mean PRT for feet was 1.26s, significantly faster than the 1.67s required for hands.
  • Impact of Distraction: While smartphone use drastically increased hand PRT to 2.71s, foot reaction time remained relatively stable at 1.42s. This suggests that while distraction severely impairs steering readiness, the reflexive "slamming on the brakes" remains relatively intact.
  • Dominance of Vehicle Speed: Statistical analysis revealed that vehicle speed at the moment of the alarm was the most critical predictor of a crash. While PRT and distance to the obstacle are factors, speed showed the clearest clustering in "crash" vs. "no crash" outcomes.

Conclusion and Future Directions

The LEE simulator demonstrates that high-fidelity research into autonomous vehicle safety is possible without multi-million-dollar investments. By using optimized game engines and consumer hardware, researchers can generate statistically significant data on human-vehicle interaction.

Future work aims to expand the variety of simulated scenarios and use LEE to generate datasets for training AI models. These models could monitor driver states in real-time to predict PRT and dynamically adjust takeover warnings based on both internal driver posture and external environmental factors.

Summary updated

BibTeX
@inproceedings{minhas2016lee,
  author = {Saad Minhas and Aura Hernández-Sabaté and Shoaib Ehsan and Katerine Díaz-Chito and Ales Leonardis and Antonio M. López and Klaus D. McDonald-Maier},
  title = {{LEE: A Photorealistic Virtual Environment for Assessing Driver-Vehicle Interactions in Self-driving Mode}},
  booktitle = {Workshops @ ECCV},
  year = {2016},
  doi = {10.1007/978-3-319-49409-8\_73},
}
Figure