Abstract
Fatigue detection is valued for people to keep mental health and prevent safety accidents. However, detecting facial fatigue, especially mild fatigue in the real world via machine vision is still a challenging issue due to lack of non-lab dataset and well-defined algorithms. In order to improve the detection capability on facial fatigue that can be used widely in daily life, this paper provided an audiovisual dataset named DLFD (daily-life fatigue dataset) which reflected people's facial fatigue state in the wild. A framework using 3D-ResNet along with non-local attention mechanism was training for extraction of local and long-range features in spatial and temporal dimensions. Then, a compacted loss function combining mean squared error and cross-entropy was designed to predict both continuous and categorical fatigue degrees. Our proposed framework has reached an average accuracy of 90.8% on validation set and 72.5% on test set for binary classification, standing a good position compared to other state-of-the-art methods. The analysis of feature map visualization revealed that our framework captured facial dynamics and attempted to build a connection with fatigue state. Our experimental results in multiple metrics proved that our framework captured some typical, micro and dynamic facial features along spatiotemporal dimensions, contributing to the mild fatigue detection in the wild.
Introduction
Most existing fatigue detection systems are designed for extreme scenarios, such as a driver falling asleep at the wheel. Consequently, the datasets used to train these models (like NTHU-DDD or YawDD) are often recorded in highly controlled lab environments or vehicle simulators.
However, "mild" fatigue—the kind experienced during a long workday—is much more subtle. In real-world ("in the wild") conditions, lighting changes, head movements are erratic, and facial cues for fatigue are not always as obvious as a massive yawn. This research addresses the lack of diverse, real-world datasets and the algorithmic difficulty of capturing subtle, long-range temporal facial dynamics, such as the specific rhythm of blinking, that signal mental exhaustion.
Methodology
The proposed end-to-end architecture is based on a 3D-ResNet-18 backbone. To handle video data, the researchers utilized a "kernel inflation" technique, expanding 2D convolutional filters into 3D to capture spatiotemporal features simultaneously.
A critical component of this architecture is the Non-local Attention Module. While standard convolutions excel at extracting local features, they struggle with long-range dependencies. Fatigue is often expressed through actions spanning many frames, such as a slow blink or subtle changes in muscle tension. The non-local block allows the model to compute the relationship between any two points in the video volume, regardless of their distance in space or time.
Finally, the framework implements a compacted loss function combining Mean Squared Error (MSE) and Cross-Entropy (CE). Using an "expectation transformation" strategy, the model outputs a categorical classification (e.g., "Alert" vs. "Fatigued") while simultaneously refining its understanding of fatigue as a continuous spectrum from 0 to 1.
The Daily-Life Fatigue Dataset (DLFD)
To support this research, the authors introduced the Daily-Life Fatigue Dataset (DLFD), which consists of 375 videos from 32 volunteers. These videos were self-captured in various real-world scenarios—from waking up to working and taking breaks—throughout the day. Unlike simulated datasets, DLFD captures natural rather than faked fatigue, providing a more rigorous benchmark for daily-use applications. Three psychology-based annotators rated the videos on a scale of 1 to 5 to provide ground truth labels.
Results and Discussion
The framework achieved a 90.8% average accuracy on the validation set and 72.5% on a more challenging test set for binary classification. It maintained an Area Under the Curve (AUC) of approximately 0.93, indicating strong discriminative power.
Key insights from the study include:
- Temporal Context: 3D models significantly outperformed 2D baselines, confirming that fatigue is defined by how facial expressions evolve over several seconds rather than single frames.
- Attention Placement: Ablation studies showed that placing the non-local attention block after the third residual block yielded the best results, allowing the model to grasp larger-scope information without early overfitting.
- Motion Dynamics: Grad-CAM visualization revealed that the network's attention focused on regions with constant motion. For alert individuals, the focus was on rapid eye blinks; for fatigued individuals with "sticky" eyelids, the focus shifted toward the mouth or other facial muscles.
Conclusion
While the model shows strong performance, the authors acknowledge that label subjectivity remains a challenge due to the internal nature of fatigue. Future work will focus on expanding the DLFD dataset to improve generalization and refining the model's sensitivity to extremely mild fatigue. Additionally, the team is working on optimizing the algorithm for mobile deployment, potentially enabling consumer applications for mental health monitoring and workplace safety.
Summary updated