AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception

Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, Yan Wang, Jing Liu, Peixuan Zhang, Peng Zhai, Lihua Zhang

Driver distraction remains a leading cause of traffic accidents globally. Despite advancements, most modern Driver Monitoring Systems (DMS) suffer from "tunnel vision," focusing exclusively on the driver's face or body while ignoring the external environment. The research paper "AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception" introduces a comprehensive dataset designed to integrate internal driver states with external traffic context for a more holistic approach to road safety.

The AIDE Dataset: A Multi-Tasking Benchmark

The primary contribution of this work is the AIDE (AssIstive Driving pErception) dataset. Unlike previous benchmarks that focus solely on endogenous factors (driver-only states), AIDE captures the interplay between the driver and exogenous factors (the traffic scene).

Overview of the AIDE dataset, showing the four primary task categories: Driver Emotion, Driver Behavior, Traffic Context, and Vehicle Condition.

The dataset consists of 2,898 samples (approximately 522K frames) of 3-second video clips. It features synchronized multi-view data from four cameras (front, left, right, and inside) and provides multi-modal annotations, including bounding boxes and keypoints for face, body (26 keypoints), and gestures (42 keypoints). AIDE is the first dataset to simultaneously address four pragmatic driving tasks:

  1. Driver Behavior Recognition (DBR): Identifying actions like smoking or using a phone.
  2. Driver Emotion Recognition (DER): Detecting states such as anger, anxiety, or weariness.
  3. Traffic Context Recognition (TCR): Understanding road conditions like traffic jams or smooth flow.
  4. Vehicle Condition Recognition (VCR): Monitoring maneuvers like turning or changing lanes.

Methodology and Fusion Strategies

To establish performance baselines, the authors evaluated three modeling patterns:

  • 2D Pattern: Standard image-based recognition using ConvNets (e.g., ResNet).
  • 2D + Timing Pattern: Combining CNNs with Transformer Encoders to capture temporal dynamics.
  • 3D Pattern: Utilizing spatio-temporal models (e.g., TimeSFormer, SlowFast) to model video representations directly.

A key technical innovation is the use of advanced fusion strategies. An Adaptive Fusion Module dynamically weights different modalities based on their situational importance, while a Cross-attention Fusion Module helps the model learn the interactions between the driver's internal state and the external scene.

Key Insights and Results

The study reveals that context is vital for accurate perception. Adding scene context significantly improves the recognition of driver behavior, as external events often explain internal reactions. For example, a driver "looking around" may be navigating a complex intersection rather than being distracted.

An example of the 'Weariness' category from the Driver Emotion Recognition task, captured by the interior camera.

Key findings include:

  • Temporal Dynamics: Spatio-temporal models (3D and 2D+Timing) consistently outperformed static 2D models, highlighting that the "flow" of a gesture or maneuver is critical for classification.
  • Beyond the Face: Body posture and gestures often provide more reliable clues for emotion and behavior recognition than facial expressions alone in naturalistic driving settings.
  • Task Synergy: There is a strong statistical correlation between driver emotion and behavior; negative states like anxiety often lead to secondary distracted behaviors.

In benchmark tests, Traffic Context Recognition (TCR) achieved the highest accuracy (92.12%), while Driver Emotion (DER) and Behavior (DBR) proved more challenging, peaking at approximately 76.52% and 78.16%, respectively.

Limitations and Future Directions

A significant challenge identified is data imbalance. In naturalistic settings, safe driving behaviors dominate the dataset, creating a "long-tail" distribution that makes it difficult for models to learn rare but critical events like "dozing off." Future research will focus on mining causal links between traffic dynamics and driver reactions, as well as developing resource-efficient architectures for real-time deployment on vehicle edge hardware.

Summary updated

Abstract

Driver distraction has become a significant cause of severe traffic accidents over the past decade. Despite the growing development of vision-driven driver monitoring systems, the lack of comprehensive perception datasets restricts road safety and traffic security. In this paper, we present an AssIstive Driving pErception dataset (AIDE) that considers context information both inside and outside the vehicle in naturalistic scenarios. AIDE facilitates holistic driver monitoring through three distinctive characteristics, including multi-view settings of driver and scene, multi-modal annotations of face, body, posture, and gesture, and four pragmatic task designs for driving understanding. To thoroughly explore AIDE, we provide experimental benchmarks on three kinds of baseline frameworks via extensive methods. Moreover, two fusion strategies are introduced to give new insights into learning effective multi-stream/modal representations. We also systematically investigate the importance and rationality of the key components in AIDE and benchmarks. The project link is https://github.com/ydk122024/AIDE.
BibTeX
@inproceedings{yang2023aide,
  author = {Dingkang Yang and Shuai Huang and Zhi Xu and Zhenpeng Li and Shunli Wang and Mingcheng Li and Yuzheng Wang and Yang Liu and Kun Yang and Zhaoyu Chen and Yan Wang and Jing Liu and Peixuan Zhang and Peng Zhai and Lihua Zhang},
  title = {{AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure