TL;DR
This paper introduces DriverMVT, a novel in-cabin dataset for driver monitoring that combines video with vehicle telemetry. This comprehensive dataset aims to advance the development of AI systems that can accurately assess a driver's state, including health, mental state, and activity, thereby enhancing road safety.
What Problem Does This Paper Address?
Road accidents are a major cause of death globally, with human error being a primary contributing factor. As automated driving systems become more prevalent, the need for robust driver monitoring systems that can assess a driver's ability to take control of the vehicle at any moment is critical. Existing driver monitoring research often focuses on specific aspects like head pose or gaze, or relies on data collected in simulated environments. There's a need for a comprehensive, real-world dataset that captures multiple facets of driver state simultaneously, including physiological signals and vehicle dynamics, to train and evaluate advanced AI models for real-time driver assessment.
Methods
The paper introduces the DriverMVT dataset, which was collected using a combination of devices: USB cameras (ELP, Samsung Galaxy S10/S20) for video capture and a Xiaomi Mi Band 3 for heart rate monitoring. The data collection involved 9 diverse drivers in real driving scenarios, capturing various conditions of speed and lighting. The dataset is structured into three categories based on synchronization precision: imprecise synchronization, precise synchronization with heart rate, and precise synchronization without heart rate. Metadata for each video includes geographic coordinates, speed, acceleration, light levels, head pose angles, gyroscope and accelerometer data, mouth openness ratio, seat belt status, and heart rate.
For evaluation, the authors employed the Multi-Task Temporal Shift Convolutional Attention Network (MTTS-CAN) algorithm, a state-of-the-art approach for heart rate estimation. They also explored a method for respiratory rate estimation using OpenPose for chest keypoint detection and optical flow for displacement analysis, particularly effective at low vehicle speeds.
Datasets
The primary contribution of this paper is the introduction of the DriverMVT dataset. This dataset comprises 1506 videos totaling over 36 hours of driving footage from 9 participants. It includes frame-by-frame annotations for:
- Head pose
- Heart rate
- Driver behavior (drowsiness, unfastened belt)
- Vehicle telemetry (speed, acceleration, light level, etc.)
The paper also references and compares DriverMVT to existing datasets such as DrivFace, MPIIGaze, DriveAHead, DMD, and MMSE-HR, highlighting DriverMVT's unique combination of real-world driving, multi-modal data, and detailed annotations.
Key Insights and Ideas
- Holistic Driver State Assessment: The DriverMVT dataset enables a more holistic approach to driver monitoring by integrating video, physiological signals (heart rate), and vehicle telemetry, moving beyond single-task datasets.
- Real-World Data is Crucial: Collecting data in actual driving scenarios, with varying speeds and lighting, is essential for developing models that generalize well to real-world conditions, unlike simulations.
- Interplay of Physiological Signals: The study suggests a direct relationship between heart rate and respiratory rate, indicating that these physiological signals can be used in conjunction for more robust driver state assessment.
- Low-Speed Respiratory Rate Estimation: The research demonstrates the feasibility of estimating respiratory rate even at very low vehicle speeds, opening avenues for monitoring during traffic jams or stop-and-go situations.
Key Results
- The DriverMVT dataset contains 1506 videos, 5119k frames, and over 36 hours of driving data.
- The MTTS-CAN algorithm, when applied to a subset of DriverMVT for heart rate estimation, achieved a mean average error of 16.375 heartbeats per minute and a root mean square error of 19.495. The authors note this as a "high error," suggesting room for improvement.
- The respiratory rate estimation algorithm was found to be effective when vehicle speed is less than 3 Km/h.
- A positive correlation was observed between heart rate and respiratory rate, with higher heart rates corresponding to higher respiratory rates.
Limitations and Future Work
- Heart Rate Estimation Error: The authors acknowledge that the mean average error of 16.375 bpm for heart rate estimation using MTTS-CAN is considered high. This suggests that current algorithms may struggle with the variability and noise present in in-cabin video data for precise heart rate measurement.
- Dataset Size for Specific Tasks: While extensive, the dataset might still be limited for training highly specialized models for very rare events or for achieving extremely high accuracy across all tasks simultaneously.
- Future Work: The authors express hope that other researchers will utilize the DriverMVT dataset to develop innovative models for driver monitoring, ultimately contributing to saving lives on the road. They envision this dataset as a foundation for future research in contactless real-time monitoring systems.
Summary updated