Drive&Act: A Multi-Modal Dataset for Fine-Grained Driver Behavior Recognition in Autonomous Vehicles

Manuel Martin, Alina Roitberg, Monica Haurilet, Matthias Horne, Simon Reiß, Michael Voit, Rainer Stiefelhagen

Introduction

As the automotive industry progresses toward higher levels of automation, the focus of computer vision research is expanding from the vehicle's exterior environment to its interior cabin. While detecting pedestrians and lane markings is critical for navigation, understanding driver behavior is essential for safety and comfort. In autonomous settings, drivers frequently engage in secondary activities—such as reading, working on laptops, or eating—which can significantly impact their readiness to resume control during emergencies.

Drive&Act addresses this need by providing a large-scale, multi-modal benchmark for fine-grained driver behavior recognition. By capturing complex interactions within the vehicle cabin, this dataset enables the development of systems capable of monitoring driver state and adapting the human-machine interface (HMI) accordingly.

The Drive&Act Dataset

The primary contribution of this work is the Drive&Act dataset, which significantly surpasses previous efforts in scale and granularity. It comprises 9.6 million frames across 12 hours of video, featuring 15 subjects performing a wide array of activities.

Comparison of Drive&Act with existing driver activity recognition datasets.

As shown in the comparison above, Drive&Act is uniquely positioned. Unlike general action recognition datasets like Kinetics, it is domain-specific. Compared to existing driver datasets like Brain4Cars or HEH, it offers more synchronized views (six), more modalities (RGB, IR, Depth, and 3D Skeletons), and a much larger number of classes (83). Furthermore, it is the only dataset in its category to provide multi-level hierarchical annotations and object-level labels.

Methodology and Hierarchical Labeling

Data was collected using a static Audi A3 driving simulator. To ensure robust recognition across varying lighting conditions, the authors utilized five near-infrared (NIR) cameras and a Microsoft Kinect v2 sensor. This setup provided a comprehensive view of the cabin through RGB, IR, and Depth modalities.

Overview of the Drive&Act dataset structure, including multi-modal views, 3D skeleton tracking, and the three-level label hierarchy.

A key innovation of Drive&Act is its hierarchical labeling scheme, which categorizes behavior into three distinct levels:

  1. Level 1 (Scenarios): Long-term tasks such as "Working on laptop."
  2. Level 2 (Fine-grained Activities): Specific actions like "Opening laptop."
  3. Level 3 (Atomic Action Units): Triplets consisting of an action, an object, and a location (e.g., "Reaching for" + "Laptop" + "Backseat").

This granularity allows for a deeper analysis of model performance, helping researchers identify whether a system struggles with object recognition or motion dynamics. Additionally, 3D upper-body skeletons were generated using OpenPose and triangulation, complemented by a "Car-Interior stream" that models the spatial relationship between the driver's joints and vehicle components like the steering wheel or gear stick.

Benchmarking and Key Results

The authors established benchmarks using both end-to-end CNN-based models (e.g., I3D, P3D) and pose-based LSTM architectures.

  • Top Performance: The Inflated 3D ConvNet (I3D) emerged as the strongest performer for fine-grained activity recognition, achieving an accuracy of 63.64%.
  • Modality Synergy: While Kinect IR data provided the best single-view results (64.98%), fusing all available views and modalities pushed the overall performance to 69.03%.
  • Pose vs. Pixels: CNNs generally outperformed pose-based models on overall activity recognition. However, pose-based models proved superior at identifying the specific location of an action, demonstrating the value of structural spatial information.
  • Illumination Invariance: The use of NIR and IR data proved critical, ensuring consistent performance regardless of external lighting conditions—a prerequisite for real-world night driving applications.

Conclusion

Drive&Act establishes a new standard for driver monitoring research. By providing a massive, multi-modal, and hierarchically labeled dataset, it challenges the community to move beyond simple action recognition toward a holistic understanding of human behavior in autonomous environments. Future work will need to address the "cross-view challenge"—maintaining accuracy when camera placements change—and extend temporal modeling to better recognize long-term scenarios.

Summary updated

Abstract

We introduce the novel domain-specific Drive&Act benchmark for fine-grained categorization of driver behavior. Our dataset features twelve hours and over 9.6 million frames of people engaged in distractive activities during both, manual and automated driving. We capture color, infrared, depth and 3D body pose information from six views and densely label the videos with a hierarchical annotation scheme, resulting in 83 categories. The key challenges of our dataset are: (1) recognition of fine-grained behavior inside the vehicle cabin; (2) multi-modal activity recognition, focusing on diverse data streams; and (3) a cross view recognition benchmark, where a model handles data from an unfamiliar domain, as sensor type and placement in the cabin can change between vehicles. Finally, we provide challenging benchmarks by adopting prominent methods for video- and body pose-based action recognition.
BibTeX
@inproceedings{martin2019drive,
  author = {Manuel Martin and Alina Roitberg and Monica Haurilet and Matthias Horne and Simon Reiß and Michael Voit and Rainer Stiefelhagen},
  title = {{Drive\&Act: A Multi-Modal Dataset for Fine-Grained Driver Behavior Recognition in Autonomous Vehicles}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00289},
}
Figure