Introduction
As the automotive industry progresses toward higher levels of automation, the focus of computer vision research is expanding from the vehicle's exterior environment to its interior cabin. While detecting pedestrians and lane markings is critical for navigation, understanding driver behavior is essential for safety and comfort. In autonomous settings, drivers frequently engage in secondary activities—such as reading, working on laptops, or eating—which can significantly impact their readiness to resume control during emergencies.
Drive&Act addresses this need by providing a large-scale, multi-modal benchmark for fine-grained driver behavior recognition. By capturing complex interactions within the vehicle cabin, this dataset enables the development of systems capable of monitoring driver state and adapting the human-machine interface (HMI) accordingly.
The Drive&Act Dataset
The primary contribution of this work is the Drive&Act dataset, which significantly surpasses previous efforts in scale and granularity. It comprises 9.6 million frames across 12 hours of video, featuring 15 subjects performing a wide array of activities.
As shown in the comparison above, Drive&Act is uniquely positioned. Unlike general action recognition datasets like Kinetics, it is domain-specific. Compared to existing driver datasets like Brain4Cars or HEH, it offers more synchronized views (six), more modalities (RGB, IR, Depth, and 3D Skeletons), and a much larger number of classes (83). Furthermore, it is the only dataset in its category to provide multi-level hierarchical annotations and object-level labels.
Methodology and Hierarchical Labeling
Data was collected using a static Audi A3 driving simulator. To ensure robust recognition across varying lighting conditions, the authors utilized five near-infrared (NIR) cameras and a Microsoft Kinect v2 sensor. This setup provided a comprehensive view of the cabin through RGB, IR, and Depth modalities.
A key innovation of Drive&Act is its hierarchical labeling scheme, which categorizes behavior into three distinct levels:
- Level 1 (Scenarios): Long-term tasks such as "Working on laptop."
- Level 2 (Fine-grained Activities): Specific actions like "Opening laptop."
- Level 3 (Atomic Action Units): Triplets consisting of an action, an object, and a location (e.g., "Reaching for" + "Laptop" + "Backseat").
This granularity allows for a deeper analysis of model performance, helping researchers identify whether a system struggles with object recognition or motion dynamics. Additionally, 3D upper-body skeletons were generated using OpenPose and triangulation, complemented by a "Car-Interior stream" that models the spatial relationship between the driver's joints and vehicle components like the steering wheel or gear stick.
Benchmarking and Key Results
The authors established benchmarks using both end-to-end CNN-based models (e.g., I3D, P3D) and pose-based LSTM architectures.
- Top Performance: The Inflated 3D ConvNet (I3D) emerged as the strongest performer for fine-grained activity recognition, achieving an accuracy of 63.64%.
- Modality Synergy: While Kinect IR data provided the best single-view results (64.98%), fusing all available views and modalities pushed the overall performance to 69.03%.
- Pose vs. Pixels: CNNs generally outperformed pose-based models on overall activity recognition. However, pose-based models proved superior at identifying the specific location of an action, demonstrating the value of structural spatial information.
- Illumination Invariance: The use of NIR and IR data proved critical, ensuring consistent performance regardless of external lighting conditions—a prerequisite for real-world night driving applications.
Conclusion
Drive&Act establishes a new standard for driver monitoring research. By providing a massive, multi-modal, and hierarchically labeled dataset, it challenges the community to move beyond simple action recognition toward a holistic understanding of human behavior in autonomous environments. Future work will need to address the "cross-view challenge"—maintaining accuracy when camera placements change—and extend temporal modeling to better recognize long-term scenarios.
Summary updated