Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling\n with Real-World Data

Akshay Rangesh, Nachiket Deo, Ross Greer, Pujitha Gunaratne, Mohan M. Trivedi

Abstract: With increasing automation in passenger vehicles, the study of safe and smooth occupant-vehicle interaction and control transitions is key. In this study, we focus on the development of contextual, semantically meaningful representations of the driver state, which can then be used to determine the appropriate timing and conditions for transfer of control between driver and vehicle. To this end, we conduct a large-scale real-world controlled data study where participants are instructed to take-over control from an autonomous agent under different driving conditions while engaged in a variety of distracting activities. These take-over events are captured using multiple driver-facing cameras, which when labelled result in a dataset of control transitions and their corresponding take-over times (TOTs). We then develop and train TOT models that operate sequentially on mid to high-level features produced by computer vision algorithms operating on different driver-facing camera views. The proposed TOT model produces continuous predictions of take-over times without delay, and shows promising qualitative and quantitative results in complex real-world scenarios.

Introduction: The Take-Over Challenge

In Level 2 and Level 3 autonomous systems, the "human-in-the-loop" remains a critical safety component. When the vehicle encounters a situation it cannot handle, it issues a Take-Over Request (TOR). However, if the driver is engaged in secondary tasks (like texting or reading), they cannot resume control instantaneously.

The core challenge is that most existing research relies on driving simulators, which often fail to capture the nuanced, "gray area" behaviors of real-world driving (e.g., a foot hovering over a pedal vs. resting on the floorboard). Furthermore, existing models often treat take-over as a single event rather than a sequence of physical actions. This paper seeks to model Take-Over Time (TOT) using high-fidelity, real-world data to determine if a safe hand-over is even possible given the current Time-to-Collision (TTC).

System overview showing the logic for issuing a take-over request based on predicted Take-Over Time (TOT) and Time-to-Collision (TTC).

Methodology: Independent LSTMs (ID-LSTMs)

The authors define the "assumption of control" as the simultaneous completion of three behaviors: Hands-on-wheel, Foot-on-pedal, and Eyes-on-road. To model this, they proposed the ID-LSTMs (Independent LSTMs) architecture, which processes driver state through several stages:

  1. Feature Extraction (Block A): Computer vision algorithms process feeds from three driver-facing cameras to produce "sensor-agnostic" features, such as gaze direction, hand locations, handheld object detection (phones, tablets), and foot activity.
  2. Parallel Processing (Block B): Instead of one large model, the system uses three separate LSTM networks to process these features. This is a key innovation: different driver behaviors have different "temporal scales." For example, eyes move and refocus much faster than a driver can move their hands from a book to the steering wheel.
  3. Continuous Prediction (Blocks C & D): The model outputs continuous predictions for each of the three components, allowing the vehicle's logic to decide in real-time whether to hand over control or initiate an emergency stop.
The ID-LSTMs architecture: (A) Feature extraction from multiple camera views, (B) Parallel LSTM processing, and (C/D) Continuous prediction of readiness components.

The Controlled Data Study (CDS) Dataset

The study introduces the CDS dataset, a large-scale real-world collection:

  • Scale: 89 subjects driving a Tesla Model S testbed.
  • Data: 1,375 take-over events captured via three cameras (face, hands, and footwell).
  • Diversity: Drivers were recorded performing eight secondary activities, including texting, reading a magazine, counting coins, and interacting with the infotainment system.
  • Uniqueness: Unlike simulator datasets, this captures naturalistic movements and real-world lighting/vibration conditions.

Key Insights and Results

The research yielded several critical findings regarding driver readiness:

  • Hierarchy of Features: Hand-related features (position and objects held) were found to be the most critical predictors of TOT. If a driver is holding a physical object like a book or phone, the time to reach the wheel increases significantly.
  • Temporal Decoupling: Processing eyes, hands, and feet through independent LSTMs outperformed a single, unified LSTM. This confirms that human physical readiness is not a monolithic state but a collection of asynchronous physical movements.
  • The "Attention Shift" Delay: The data showed that even for tasks not involving the feet (like texting), foot-on-pedal times increased. This suggests a cognitive "switching cost" where the brain must finish processing the secondary task before initiating any physical recovery movement.
  • Accuracy: The ID-LSTMs model achieved a Mean Absolute Error (MAE) of ~0.79 seconds for total TOT. It successfully captured task-specific variance, noting that "attentive" drivers have very low TOT, while tasks like "reading" or "texting" significantly push TOT higher.

Discussion and Future Work

While the model shows promise, several areas for future development remain. The "eyes closed" test cases used simulated drowsiness, which may not fully represent the cognitive lag of real sleep. Additionally, while the paper proposes the logic $TOT + \epsilon < TTC$, the study focuses primarily on the $TOT$ prediction. Future work should integrate this with real-time external scene analysis to trigger varied vehicle responses. Finally, as the data was collected in a Tesla Model S, different cabin geometries might require model recalibration to maintain generalization.

Summary updated

Abstract

With increasing automation in passenger vehicles, the study of safe and\nsmooth occupant-vehicle interaction and control transitions is key. In this\nstudy, we focus on the development of contextual, semantically meaningful\nrepresentations of the driver state, which can then be used to determine the\nappropriate timing and conditions for transfer of control between driver and\nvehicle. To this end, we conduct a large-scale real-world controlled data study\nwhere participants are instructed to take-over control from an autonomous agent\nunder different driving conditions while engaged in a variety of distracting\nactivities. These take-over events are captured using multiple driver-facing\ncameras, which when labelled result in a dataset of control transitions and\ntheir corresponding take-over times (TOTs). We then develop and train TOT\nmodels that operate sequentially on mid to high-level features produced by\ncomputer vision algorithms operating on different driver-facing camera views.\nThe proposed TOT model produces continuous predictions of take-over times\nwithout delay, and shows promising qualitative and quantitative results in\ncomplex real-world scenarios.\n
BibTeX
@inproceedings{rangesh2021autonomous,
  author = {Akshay Rangesh and Nachiket Deo and Ross Greer and Pujitha Gunaratne and Mohan M. Trivedi},
  title = {{Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling\textbackslash\{\}n with Real-World Data}},
  booktitle = {ITSC},
  year = {2021},
  doi = {10.1109/itsc48978.2021.9564434},
}
Figure