TL;DR
This paper introduces AFCN, a novel network for facial emotion recognition (FER) that tackles the pervasive issue of label noise in large datasets. By incorporating attention mechanisms and consistency learning, AFCN effectively mitigates the impact of noisy labels and improves the model's ability to capture global facial features, leading to state-of-the-art performance on challenging benchmarks.
What Problem Does This Paper Address?
Facial expression recognition (FER) is crucial for understanding human emotions and has applications in fields like education. However, a significant challenge in FER is the presence of "label noise" in large datasets. This noise arises from the inherent ambiguity of facial expressions (e.g., subtle differences between disgust and sadness) and subjective biases of human annotators. Existing FER models are highly susceptible to this noise, leading to incorrect feature learning, overfitting, and ultimately, reduced accuracy. While some methods attempt to address label noise through sample selection or label correction, they often overlook the tendency of models to rely too heavily on localized facial features, which can be misleading when those local features are mislabeled. This paper aims to develop a robust FER model that can effectively handle noisy labels by focusing on both local and global facial cues and ensuring consistency in its predictions.
Methods
The proposed AFCN (Attention-Based Fusion Consistency Network) is built upon four key modules:
- Sample Certainty Analysis Module: This module quantifies the "certainty" or dependability of each input facial image's prediction. It uses a sigmoid function to generate certainty weights (between 0 and 1) based on deep features extracted by a backbone network (ResNet50). A regularization loss is introduced to ensure that high-certainty samples are distinctly separated from low-certainty ones.
- Label Correction Module: For samples identified as having low certainty, this module attempts to correct their labels. It compares the model's maximum predicted probability with the original label's probability. If the former significantly exceeds the latter, the label is updated to the model's prediction.
- Attention Fusion Module: This module leverages Grad-CAM to generate attention maps for each facial expression category. These maps highlight the crucial local regions the model focuses on. The module then fuses these individual attention maps to create a "merged" attention map that encompasses all potential key regions for an expression.
- Fusion Consistency Learning Module: This module enforces consistency between the model's attention on the "ground truth" regions (based on the potentially corrected label) and the fused attention map of all potential key regions. This encourages the model to learn global facial features rather than relying solely on potentially noisy local cues. Additionally, a "mutual exclusivity" loss is introduced to ensure that attention maps for different emotion categories are distinct, preventing the model from collapsing into a single attention point.
The overall loss function is a weighted sum of the logit-weighted cross-entropy loss (for classification), the sample certainty regularization loss, the fusion consistency loss, and the mutual exclusivity loss.
Datasets
The experiments were conducted on three publicly available facial expression recognition datasets:
- RAF-DB (Real-world Affective Faces Database): A large-scale dataset commonly used for FER.
- FERPlus (Facial Expression Recognition Plus): An extension of the FER2013 dataset.
- AffectNet: Another large-scale dataset for facial expression analysis.
The authors introduced varying levels of label noise (10%, 20%, and 30%) to these datasets to evaluate the robustness of their method.
Key Insights and Ideas
- Label Noise is a Major Hurdle: The paper strongly emphasizes that label noise significantly degrades FER model performance, a problem that persists even with existing noise-handling techniques.
- Global vs. Local Features: A critical insight is that models often over-rely on local facial features, which are more susceptible to mislabeling in noisy datasets. AFCN's strength lies in its ability to fuse local attention maps to capture global expression patterns.
- Attention as a Diagnostic Tool: Grad-CAM is effectively used not just for visualization but as a core component to identify and fuse relevant facial regions, guiding the model towards more robust feature extraction.
- Consistency is Key: The fusion consistency loss is a novel approach to ensure that the model's understanding of an expression is consistent across all relevant facial cues, not just the most prominent (and potentially mislabeled) ones.
- Mutual Exclusivity for Discrimination: The mutual exclusivity loss is a clever addition to prevent the model from becoming overly generalized and to encourage it to learn distinct representations for different emotion categories.
Key Results
- State-of-the-Art Performance: AFCN consistently outperforms existing state-of-the-art methods (SCN, RUL, EAC) on the RAF-DB and FERPlus datasets across different noise levels (10%, 20%, 30%).
- Significant Improvement on Noisy Data: Notably, on the RAF-DB dataset with 30% noise, AFCN achieved a 3.03% accuracy improvement over the best compared method.
- Robustness to Noise: The confusion matrices demonstrate AFCN's ability to accurately recognize various expressions even with high levels of label noise.
- Ablation Study Validation: An ablation study confirmed the effectiveness of each proposed module, with the full AFCN model (including all modules) achieving the best performance.
- Efficiency: The use of ResNet50 as a backbone makes the model compatible with low-computational-power environments, achieving 400 frames per second on an RTX 3090 GPU, suggesting potential for real-time applications.
Limitations and Future Work
- AffectNet Performance Nuances: While AFCN performs well on AffectNet, it slightly underperforms compared to some methods at lower noise levels (10% and 20%). The authors suggest this might be due to AffectNet's complexity, involving factors beyond label noise.
- Extreme Real-World Scenarios: The paper acknowledges that performance in extreme real-world scenarios, such as severe occlusions, complex backgrounds, or ultra-low-power devices, requires further investigation.
- Hyperparameter Tuning: While the paper explores parameter sensitivity for $\lambda_3$ and $\lambda_4$, further optimization and analysis of other hyperparameters could be beneficial.
The authors suggest that future work should focus on ensuring robust deployment in diverse real-world scenarios and exploring its capabilities on ultra-low-power devices.
Summary updated