Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild

Guangyao Zhou, Yuanlun Xie, Wenhong TianarXiv 2023 · OpenReview

TL;DR

This paper tackles the challenging problem of facial expression recognition (FER) in real-world, unconstrained "in the wild" scenarios. The authors introduce a novel single model (R18+FAML) that combines internal feature fusion and attention mechanisms with multiple loss functions, and an ensemble strategy (R18+FAML-FGA-T2V) that further fuses features from multiple networks and employs a "Top Two Voting" approach. These methods achieve state-of-the-art results on challenging wild datasets.

What Problem Does This Paper Address?

Facial Expression Recognition (FER) is crucial for human-computer interaction, but accurately recognizing emotions from images captured in the wild is significantly harder than in controlled lab settings. This difficulty stems from factors like varying illumination, occlusions (e.g., hands covering parts of the face), subtle differences between expressions (inter-class similarity), and imbalanced datasets where some emotions are much more common than others. Existing deep learning models, while powerful, often struggle with these real-world complexities, leading to suboptimal performance. Furthermore, most prior work focuses on either single model improvements or ensemble methods, but rarely combines advanced feature fusion techniques with sophisticated ensemble strategies for wild FER.

Methods

The paper proposes a multi-pronged approach to enhance FER performance:

  • R18+FAML (Single Model): This model builds upon the ResNet18 architecture. Key innovations include:
  • Internal Feature Fusion: Modifying the initial convolutional layers to fuse features from different image parts, aiming to amplify important facial cues.
  • Attention Blocks: Incorporating Vision Transformer (ViT) based attention blocks within the ResNet architecture to focus on salient local features.
  • Multiple Loss Functions (FAML): Employing a combination of Label Smoothing Regularization (LSR) and Cross-Entropy (CE) loss. This multi-loss strategy encourages the model to explore diverse feature extraction paths, potentially avoiding local optima that single loss functions might get stuck in.
  • Learning Rate Strategies: Using two distinct learning rate schedules (Cosine Annealing and StepLR) to optimize training.
  • R18+FAML-FGA-T2V (Ensemble Model): This model builds upon R18+FAML by adding two crucial components:
  • Feature Fusion among Networks (FGA): A Genetic Algorithm (GA) based approach to fuse the convolution kernel parameters from multiple well-trained networks of the same architecture. This aims to inherit the best feature extraction capabilities from various models.
  • Top Two Voting (T2V) Ensemble Strategy: Instead of just considering the top predicted class, T2V incorporates the second-highest predicted class from each network into the final decision. This is motivated by observations that the second-ranked prediction often contains valuable information, especially when networks are "hesitant" between similar expressions. This strategy directly influences classification without retraining the base models.

Datasets

The models were trained and evaluated on three challenging "in the wild" facial expression recognition datasets:

  • RAF-DB (Real-world Affective Faces Database): Contains 30,000 facial images with annotations for 7 basic emotions. The paper uses 12,271 images for training and 3,068 for testing.
  • AffectNet-8 and AffectNet-7: AffectNet is a large-scale dataset with over a million images. AffectNet-7 includes 7 basic expressions, while AffectNet-8 adds "Contempt." The paper uses 283,901 training and 3,500 test images for AffectNet-7, and 287,568 training and 4,000 test images for AffectNet-8.

Key Insights and Ideas

  • The "Wild" is Tough: The paper strongly emphasizes that performance gains on lab datasets do not directly translate to "in the wild" scenarios, highlighting the need for specialized approaches.
  • Multi-Loss for Diversity: Using multiple loss functions isn't just about averaging; it encourages the model to find solutions that satisfy multiple learning objectives simultaneously, leading to more robust feature extraction.
  • Ensembling for the Win (Especially T2V): The ensemble strategy, particularly the Top Two Voting (T2V), proves highly effective. The insight that the second-best prediction often holds crucial information is a key driver for its success.
  • Fusing the "Best" of Networks: The FGA approach to fuse convolution kernels from multiple networks is a novel way to leverage collective intelligence, rather than just averaging outputs.
  • Attention is Good, but Not Everything: While attention mechanisms improve performance, the paper notes that even without explicit attention blocks, their core FAML strategy can achieve high proficiency, suggesting the internal fusion and multi-loss are powerful on their own.

Key Results

The proposed models achieve state-of-the-art performance on the tested datasets:

  • Single Model (R18+FAML): Achieved accuracies of 90.32% on RAF-DB, 62.17% on AffectNet-8, and 65.83% on AffectNet-7. This is notable as it's one of the few non-ensemble models to surpass 90% on RAF-DB.
  • Ensemble Model (R18+FAML-FGA-T2V): Further improved performance to 91.59% on RAF-DB, 63.27% on AffectNet-8, and 66.63% on AffectNet-7. This represents a significant improvement over previous state-of-the-art methods, including other ensemble approaches.
  • Ablation Studies: Demonstrated that each component (FA, FAML, FGA, T2V) contributes to the overall performance, with the ensemble strategy (T2V) showing particularly strong gains.
  • Universality of T2V: The T2V strategy showed consistent improvements across various backbone architectures (ResNet, VGG, DenseNet), indicating its broad applicability.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that their focus was on achieving high accuracy, not necessarily computational efficiency. The ensemble approach, while effective, involves multiple networks.
  • Future Work: The authors plan to explore more advanced feature fusion techniques and convolution operation modes. They also aim to reduce computational complexity through methods like automatic parameter adjustment and investigate applying their strategies to other advanced models for even higher accuracy.

Summary updated

BibTeX
@misc{zhou2023multi,
  author = {Guangyao Zhou and Yuanlun Xie and Wenhong Tian},
  title = {{Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild}},
  year = {2023},
  url = {https://openreview.net/forum?id=veESZP5QLA},
}
Figure