TL;DR
This paper introduces MixAugment, a novel data augmentation technique for facial expression recognition (FER) that combines real and synthetically generated (Mixup) examples during training. MixAugment significantly improves FER performance on challenging "in-the-wild" datasets compared to standard Mixup and other state-of-the-art methods, particularly for minority classes.
What Problem Does This Paper Address?
Facial Expression Recognition (FER) is crucial for human-computer interaction, but accurately recognizing emotions in real-world scenarios is challenging due to variations in head poses, lighting, and occlusions. Deep Neural Networks (DNNs), while powerful, are prone to overfitting, especially when trained on limited or imbalanced datasets. Existing data augmentation techniques, like Mixup, help by creating virtual training examples, but they can sometimes produce unrealistic combinations when applied to highly varied "in-the-wild" data, potentially hindering DNN learning. This paper aims to address these limitations by proposing a more robust augmentation strategy.
Methods
The paper builds upon the Mixup data augmentation technique, which trains DNNs on convex combinations of pairs of examples and their labels. The authors introduce MixAugment, a novel strategy that trains the DNN concurrently on both real (original) and virtual (Mixup-generated) examples. The total loss function in MixAugment incorporates the classification loss from real examples and the Mixup loss from virtual examples. This approach aims to leverage the benefits of Mixup while mitigating the generation of unrealistic samples by keeping real data in the training loop. The authors also investigate the impact of combining MixAugment with dropout and other augmentation techniques like flipping.
Datasets
The primary dataset used for experiments is the Real-world Affective Faces Database (RAF-DB). This is a large-scale "in-the-wild" facial expression database containing approximately 30,000 images with crowdsourced annotations for seven basic expressions. The paper notes that RAF-DB exhibits significant variability in age, gender, ethnicity, head poses, lighting, and occlusions. The authors used the single-label subset of RAF-DB, splitting it into a training set of around 12,200 images and a test set of around 3,100 images. They also mention experiments using other widely used DNNs and pre-training on AffectNet.
Key Insights and Ideas
- MixAugment's Synergy: Combining real and Mixup-generated samples in the training loss (MixAugment) is more effective than solely relying on Mixup, especially for challenging "in-the-wild" data. This suggests that maintaining a direct connection to real data is crucial for robust learning.
- Addressing Unrealistic Mixups: The paper highlights that standard Mixup can create visually unrealistic facial expressions when mixing images with vastly different head poses, which can negatively impact DNN training. MixAugment's approach of using both real and virtual samples helps to overcome this.
- Dropout's Double-Edged Sword: While dropout is a common regularization technique, its combination with Mixup or MixAugment did not consistently yield the best results, and in some cases, it seemed to negatively impact performance. Optimal results were often achieved without dropout.
- Minority Class Improvement: MixAugment shows a substantial improvement in performance metrics for minority classes, such as "fearful," which is particularly valuable for imbalanced datasets. This indicates its effectiveness in boosting recognition for less represented categories.
- Faster Convergence: Training with MixAugment leads to faster convergence compared to using Mixup or no augmentation, suggesting improved training efficiency.
Key Results
- MixAugment consistently outperformed standard Mixup and vanilla training (no augmentation) across all studied metrics (Accuracy, F1-score, Average Accuracy).
- In terms of accuracy, MixAugment achieved up to 86.06% (with flipping) compared to 84.06% for Mixup and 83.31% for the vanilla case.
- The proposed MixAugment technique showed a significant increase in performance for the minority class "fearful," with Precision, Recall, and F1-score increasing by approximately 10%, 5%, and 6% respectively.
- MixAugment demonstrated superior performance compared to several state-of-the-art methods, achieving the highest accuracy on the RAF-DB dataset.
Limitations and Future Work
- Dropout Interaction: The paper notes that the optimal combination of dropout with Mixup or MixAugment was not always clear, and sometimes dropout seemed to hinder performance. Further investigation into this interaction could be beneficial.
- Hyperparameter Sensitivity: The authors observed that large values of the Mixup hyperparameter (α) could lead to underfitting. While they found optimal values for MixAugment, exploring more adaptive hyperparameter selection could be a future direction.
- Future Work: The authors plan to extend MixAugment to other small and large-scale "in-the-wild" datasets, as well as to other affect recognition tasks like valence-arousal estimation and action unit detection. They also aim to explore its application in other computer vision domains.
Summary updated