TL;DR
This paper introduces DE-ViViT, a novel Dual Embedding Video Vision Transformer for robust eye blink detection that handles varying camera angles. By combining tubelet and residual embeddings, DE-ViViT captures both macro and micro changes in eye movements, achieving state-of-the-art performance on challenging datasets and demonstrating remarkable consistency across different viewpoints.
What Problem Does This Paper Address?
Eye blink detection is a critical biomarker for assessing physical and mental states, with applications ranging from driver fatigue monitoring to diagnosing ocular conditions. While non-contact video-based methods are preferred for their non-invasiveness, existing approaches often struggle with variations in camera angles. Many models are optimized for frontal views and fail to generalize to real-world scenarios where subjects may not be directly facing the camera. This limitation is exacerbated by the small size of the eye region and the rapid nature of blinks, making robust detection a significant challenge. The paper aims to address this by developing a model that is resilient to diverse camera perspectives.
Methods
The core innovation is the Dual Embedding Video Vision Transformer (DE-ViViT). This architecture builds upon the Vision Transformer (ViT) framework and incorporates two novel embedding strategies:
- Tubelet Embedding: This component uses 3D convolutional operations to capture macro-level changes across spatio-temporal sequences. It employs overlapping strides to allow convolutional kernels to slide across neighboring pixels, extracting significant features and capturing temporal nuances.
- Residual Embedding: This technique is designed to capture subtle, micro-variations within the eye movement sequence. It works by performing a pixel-by-pixel subtraction between the original input sequence and key frames (identified as pivotal moments in a blink). This effectively isolates residual information about eyelid movement during a blink, which is crucial for rapid, subtle changes.
These two embeddings are concatenated and supplemented with positional embeddings before being fed into a standard Transformer encoder. The encoder, featuring Multi-Head Self-Attention (MSA) and an MLP block, processes the enriched sequence. Finally, a classification head predicts the probability of an eye blink. For eye region extraction, the authors utilize YOLOv8 for object detection and segmentation.
Datasets
The study utilizes two datasets:
- HUST-LEBW: This is a publicly available, in-the-wild dataset used for both training and evaluating the model. It comprises eye blink instances from 20 commercial movies, offering a variety of conditions. The paper adapted sequences to a consistent 10-frame input.
- MAEB (Multi-Angle Eye Blink Dataset): This is a custom-collected dataset specifically designed for this research. It features simultaneous captures from nine different camera angles, allowing for a detailed analysis of angle-specific performance. The dataset consists of 1,440 sequences with an equal number of blink and non-blink instances.
Key Insights and Ideas
- Dual Embedding for Robustness: The combination of tubelet (macro changes) and residual (micro changes) embeddings is key to capturing the complex dynamics of eye blinks, especially under varying conditions.
- ViT for Temporal Dynamics: Leveraging the Vision Transformer architecture, which excels at capturing long-range dependencies, proves effective for analyzing sequential eye movements.
- MAEB Dataset's Value: The MAEB dataset is a significant contribution, providing a controlled yet diverse environment to rigorously test the robustness of blink detection models against camera angle variations.
- Angle-Specific Vulnerabilities: Even with a robust model like DE-ViViT, performance can degrade at specific camera angles (e.g., corners), highlighting that "perfect" generalization remains an ongoing challenge.
Key Results
- State-of-the-Art Performance: DE-ViViT achieved the highest performance metrics on the HUST-LEBW dataset, comparable to or exceeding other advanced models like CNN-LSTM2.
- MAEB Dataset Dominance: On the MAEB dataset, DE-ViViT outperformed other baselines, demonstrating its superior ability to handle varying camera angles.
- Consistent Performance: A key finding is DE-ViViT's minimal performance variation across different camera angles on the MAEB dataset, indicating strong robustness. For instance, on MAEB, DE-ViViT achieved an F1-score of .894, significantly higher than other models.
- Statistical Significance: A One-way Repeated Measures ANOVA test confirmed statistically significant differences in detection outcomes across camera angles, but DE-ViViT showed the most consistent performance.
Limitations and Future Work
- Angle-Specific Performance Drop: While DE-ViViT is robust, performance still drops at certain camera angles (e.g., corners). Future research could focus on further enhancing robustness to these variations.
- Two-Stage Pipeline: The current pipeline involves separate eye region detection and blink identification. For real-time applications, optimizing this pipeline or exploring end-to-end solutions might be necessary, as the current processing time (15.31 ms per 10-frame sequence) doesn't include the entire input-to-output flow.
- Precise Blink Timing: The paper suggests exploring attention scores for more precise blink timing, as self-attention mechanisms inherently capture relationships between frames.
- Real-World Applications: Investigating the application of this technology in real-world scenarios like driver drowsiness detection or deception detection, where subtle facial cues are important, is a promising avenue.
- Facial Movement and Expression: Future work could also explore how facial movement and expression influence blink detection and how to incorporate this for a more comprehensive analysis.
Summary updated