TL;DR
This paper introduces EM-CNN, a novel convolutional neural network for detecting driver fatigue from images. By combining a robust face and feature point detection system (MTCNN) with a specialized eye and mouth state recognition network (EM-CNN), the approach achieves high accuracy in identifying fatigue indicators like eye closure and yawning. This work offers a promising step towards safer driving through automated fatigue monitoring.
What Problem Does This Paper Address?
Driver fatigue is a significant contributor to traffic accidents, with studies indicating it's responsible for a substantial percentage of crashes and can increase accident probability manifold. Traditional methods for detecting fatigue often rely on physiological sensors that can be uncomfortable for drivers or on vehicle state analysis that is highly dependent on individual driving habits and road conditions, leading to reduced accuracy. This paper tackles the challenge of developing an efficient, reliable, and non-invasive method for driver fatigue detection using computer vision techniques, specifically by analyzing facial expressions captured in driving images.
Methods
The proposed system is a two-stage approach:
- Face and Feature Point Detection (MTCNN): The Multi-task Cascaded Convolutional Network (MTCNN) is employed for robust face detection and the precise localization of key facial landmarks. This includes the bounding box of the face and the coordinates of the left and right eyes, nose, and mouth corners. MTCNN is a hierarchical network that refines detection from coarse to fine through three cascaded subnetworks (P-Net, R-Net, O-Net), making it suitable for the unconstrained environments of real driving scenarios.
- Eye and Mouth State Recognition (EM-CNN): A novel Convolutional Neural Network, named EM-CNN, is developed to classify the states of the eyes (open/closed) and mouth (open/closed) from the extracted Regions of Interest (ROIs). The EM-CNN architecture is designed to be adaptable to varying image sizes and lighting conditions. It utilizes multiple convolutional and pooling layers, followed by a residual block, to extract features. The final output is a classification of the eye and mouth states.
Finally, driver fatigue is assessed by combining the outputs of EM-CNN with two key metrics:
- PERCLOS (Percentage of Eyelid Closure over the Pupil over Time): Calculated based on the eye closure state. A threshold of 0.25 is used to indicate prolonged eye closure.
- POM (Mouth Opening Degree): Calculated based on the mouth opening state. A threshold of 0.5 is used to indicate prolonged mouth opening (yawning).
The paper also mentions a machine vision-based method for eye closure calculation involving binarization, smoothing, and pixel ratio analysis, which is used for defining the eye state.
Datasets
The study utilized a dataset of 4000 real driving environment images provided by an IT company. This dataset was divided into four categories:
- Open eyes (2226 images)
- Closed eyes (1774 images)
- Open mouth (1996 images)
- Closed mouth (2004 images)
The dataset was split into training and testing sets using a 3/7 ratio. The paper does not mention the creation of a new benchmark dataset.
Key Insights and Ideas
- Leveraging MTCNN for Robust Feature Extraction: The use of MTCNN is crucial for accurately detecting faces and key facial landmarks, even in challenging driving conditions. This robust initial step significantly improves the reliability of subsequent fatigue detection.
- EM-CNN's Specialization for Eye and Mouth States: The EM-CNN is specifically designed to capture subtle changes in eye and mouth states, which are primary indicators of fatigue. Its architecture is optimized to handle variations in lighting and pose.
- Combining PERCLOS and POM for Comprehensive Fatigue Assessment: Relying on a single indicator can be insufficient. The paper's approach of combining PERCLOS (eye closure) and POM (yawning) provides a more comprehensive and reliable measure of driver fatigue.
- Binocular vs. Monocular Information: The paper emphasizes the advantage of using binocular (two-eye) images over monocular ones for more accurate eye state detection, reducing the risk of misjudgment due to head shifts.
Key Results
The proposed EM-CNN model demonstrated strong performance:
- Accuracy: 93.623%
- Sensitivity: 93.643%
- Specificity: 60.882%
The EM-CNN outperformed other established CNN architectures like AlexNet, VGG-16, GoogLeNet, and ResNet50 in terms of accuracy and sensitivity. The paper also notes that EM-CNN is more sensitive to mouth states than eye states, attributing this to the more pronounced visual changes in the mouth during fatigue. Area Under the Curve (AUC) values for the ROC curves were very high across all four classifications (eye close, eye open, mouth close, mouth open), indicating excellent classification performance.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the specificity for eye states is lower compared to mouth states.
- The thresholds for PERCLOS (0.25) and POM (0.5) were determined experimentally on a limited set of video sequences and might require further validation.
- While the method is robust, the paper implies that further testing of actual performance and robustness in diverse real-world scenarios is needed.
- Future Work:
- The authors plan to further test the actual performance and robustness of the proposed method.
- A key future direction is to implement the method on a hardware device for real-time deployment.
Summary updated