TL;DR
This paper introduces Squeeze ViT, a novel Vision Transformer architecture for facial expression recognition (FER). By combining global and local visual features with facial landmark information and incorporating a "squeeze module" to reduce computational complexity, Squeeze ViT achieves state-of-the-art performance on both lab-controlled and wild FER datasets while being significantly more efficient than standard ViTs.
What Problem Does This Paper Address?
Facial expression recognition (FER) is crucial for human-computer interaction and understanding emotional states. While Convolutional Neural Networks (CNNs) have been effective, they often struggle with subtle expression changes and require extensive feature engineering. Vision Transformers (ViTs) have shown promise in image recognition due to their ability to capture global context, but their standard architecture can lose crucial local details essential for FER. Existing ViTs also suffer from high computational costs. This paper aims to bridge this gap by developing a ViT-based FER model that effectively integrates both global and local features, improves accuracy, and reduces computational burden.
Methods
The proposed Squeeze ViT model builds upon the Vision Transformer architecture. Key innovations include:
- Hybrid Feature Representation: Instead of relying solely on visual tokens, Squeeze ViT incorporates both visual tokens derived from image patches and "landmark tokens" generated from facial landmark heatmaps. This allows the model to capture both global image context and fine-grained local facial movements.
- Squeeze Module: To address the computational inefficiency of standard ViTs, a "squeeze module" is introduced. This module progressively reduces the dimensionality of feature representations across transformer encoder layers. This not only decreases the number of parameters and computations but also helps maintain discriminative features through feature-wise attention.
- Architecture: The model uses a ResNet-18 backbone to extract initial global and local features. These features are then tokenized, combined with a class token and positional embeddings, and fed into a series of transformer encoder and squeeze module layers. The final classification is performed using a prediction head on the class token.
Datasets
The paper evaluates Squeeze ViT on three datasets:
- CK+ (Cohn-Kanade Extended): A widely used lab-controlled dataset with 7 basic emotion labels.
- MMI: Another lab-controlled dataset with 6 basic facial expressions.
- RAF-DB (Real-world Affective Face Database): A challenging wild dataset with 7 basic and 11 complex emotion labels, featuring diverse conditions like lighting, pose, and occlusions.
Key Insights and Ideas
- Local Features are Crucial for FER: The study strongly emphasizes that while ViTs excel at global feature extraction, facial expressions rely heavily on subtle, local muscle movements. Integrating facial landmark information alongside visual features is essential for robust FER.
- Efficiency Gains from Squeeze Module: The squeeze module is not just about reducing parameters; it's a clever way to manage computational complexity without sacrificing performance. This is a significant step towards making powerful transformer models more practical for resource-constrained applications.
- Unified Approach for Wild and Lab Data: The model's ability to perform well on both highly controlled lab datasets and the more challenging, diverse "wild" RAF-DB dataset highlights its robustness and generalizability.
Key Results
- Accuracy Improvement: Squeeze ViT achieved state-of-the-art performance on the CK+ and MMI datasets, with an accuracy of 99.54% and 89.89% respectively. It also performed competitively on the RAF-DB dataset with 88.90% accuracy.
- Efficiency: Compared to a pure ViT, Squeeze ViT reduced the number of parameters by approximately 7.26-fold and computations by about 17.95-fold, demonstrating significant efficiency gains.
- Ablation Study: Combining visual and landmark tokens improved performance by 0.9% on the RAF-DB dataset, underscoring the importance of the hybrid feature approach.
Limitations and Future Work
- Wild Dataset Challenges: The paper acknowledges that performance on the RAF-DB dataset, a wild dataset, is slightly lower than on lab-controlled datasets. This is attributed to difficulties in accurately extracting facial landmarks due to extreme lighting, occlusions, and pose variations.
- Future Directions: The authors plan to extend their research to include feature extraction by estimating landmarks or facial patches even in occluded face regions. This would further enhance the model's robustness in real-world scenarios.
Summary updated