TL;DR
This paper introduces a novel gaze estimation model, Res-Swin-GE, that combines the strengths of Convolutional Neural Networks (CNNs) and the Swin Transformer architecture. By replacing the initial patch-embedding mechanism of the Swin Transformer with a ResNet backbone, the model significantly improves the preservation of local spatial features, leading to state-of-the-art performance on benchmark gaze estimation datasets.
What Problem Does This Paper Address?
Gaze estimation, the process of determining where a person is looking, is crucial for understanding human behavior and has applications in fields like human-computer interaction, virtual reality, and medical treatment. While Convolutional Neural Networks (CNNs) have been successful, they can struggle with capturing long-range dependencies. More recently, Transformer models, particularly the Swin Transformer, have shown promise due to their ability to model global relationships. However, the standard Swin Transformer's method of dividing images into non-overlapping patches and then processing them can lead to the loss of fine-grained local spatial information, which is critical for accurate gaze estimation. This paper aims to bridge this gap by developing a model that leverages the global modeling capabilities of Swin Transformers while preserving essential local features.
Methods
The authors propose two main network architectures:
- SwinT-GE: This is a direct application of the Swin Transformer (specifically, the "tiny" version) for gaze estimation. It uses the standard Swin Transformer's patch partitioning and linear embedding layers to process input images.
- Res-Swin-GE: This is the proposed hybrid model. It replaces the initial patch partitioning and linear embedding layers of SwinT-GE with a ResNet18 backbone. The ResNet18 is used for initial feature extraction, preserving local spatial details, and its output is then fed into the Swin Transformer blocks for further processing and global feature learning. This approach aims to mitigate the loss of local information inherent in the standard Swin Transformer's patch-based processing.
The core innovation lies in replacing the slicing-and-mapping mechanism of the Swin Transformer with a CNN (ResNet18) to better retain local spatial features before the attention mechanisms are applied.
Datasets
The proposed models were evaluated on two publicly available datasets:
- MpiiFaceGaze [38]: A popular appearance-based gaze estimation dataset collected in real-world environments, offering rich variations in lighting and head pose. It contains 213,659 facial images from 15 participants.
- Eyediap [39]: This dataset was collected using depth and RGB cameras, with gaze direction calculated from depth camera coordinates. It contains 94 video segments from 16 participants. While providing accurate gaze labels, it has less variation in lighting conditions compared to MpiiFaceGaze.
The authors followed a standard preprocessing procedure, cropping images to 224x224 pixels for gaze estimation.
Key Insights and Ideas
- The trade-off between local and global features: The paper highlights that while Transformers excel at global context, their standard input processing can sacrifice crucial local details. CNNs, on the other hand, are adept at local feature extraction but may struggle with long-range dependencies.
- Hybrid architectures are powerful: The success of Res-Swin-GE demonstrates that combining the strengths of CNNs and Transformers can lead to superior performance by leveraging the best of both worlds.
- Importance of early feature extraction: Replacing the Swin Transformer's initial patch-based embedding with a CNN backbone (ResNet18) significantly improves the model's ability to capture fine-grained eye features, which are essential for accurate gaze estimation.
- Personal calibration is a persistent challenge: The study acknowledges that individual differences in eye structure (kappa angle) lead to a "personal calibration problem," affecting model performance across different subjects. This is a known issue in the field.
Key Results
- Res-Swin-GE significantly outperforms SwinT-GE: The hybrid model achieved notably lower angular errors on both datasets. For instance, on MpiiFaceGaze, Res-Swin-GE achieved an average angular error of 3.75°, while SwinT-GE achieved 9.29°.
- State-of-the-art performance: Res-Swin-GE achieved competitive results, reaching an angular error of 3.75° on MpiiFaceGaze and 4.78° on Eyediap. It outperformed existing methods, showing a 7.5% improvement over GazeTR-Hybrid on the Eyediap dataset.
- ResNet18's contribution is crucial: Ablation studies showed that removing the ResNet18 backbone from Res-Swin-GE led to a substantial increase in angular error (e.g., from 3.75° to 9.29° on MpiiFaceGaze), underscoring its importance in preserving local features.
- Swin-T's contribution is also vital: Conversely, removing the Swin-T structure from Res-Swin-GE also increased angular errors, highlighting the necessity of its global modeling capabilities.
- Window size optimization: The paper found that an optimal window size of 2 for the Swin Transformer blocks yielded the best performance for Res-Swin-GE.
Limitations and Future Work
- Personal Calibration: The paper acknowledges the "personal calibration problem" where individual differences in eye structure impact model performance. While the proposed method shows improved stability, this remains a challenge.
- Extreme Gaze Angles: Performance degrades at extreme gaze angles, likely due to fewer training samples and the amplified effect of personal calibration issues.
- Future Work: The authors plan to focus on extracting more robust features to address individual calibration issues. They also aim to apply Res-Swin-GE to virtual and augmented reality applications for more immersive experiences.
Summary updated