Semantic Data Augmentation for Long-tailed Facial Expression Recognition

Zijian Li, Yan Wang, Bowen Guan, Jiankai Yin

TL;DR: Researchers have developed a novel semantic data augmentation method to fix the "long-tail" problem in Facial Expression Recognition (FER). By using a VAE-GAN to manipulate images in a compressed latent space—guided by class-specific statistical covariance—they can generate realistic, diverse new samples for rare emotions like "fear" or "disgust," significantly boosting model accuracy.

Abstract

Facial Expression Recognition (FER) has a wide application prospect in social robotics, health care, driver fatigue monitoring, and many other practical scenarios. Automatic recognition of facial expressions has been extensively studied by the Computer Vision research society. But Facial Expression Recognition in real-world is still a challenging task, partially due to the long-tailed distribution of the dataset. Many recent studies use data augmentation for Long-Tailed Recognition tasks. In this paper, we propose a novel semantic augmentation method. By introducing randomness into the encoding of the source data in the latent space of VAE-GAN, new samples are generated. Then, for facial expression recognition in RAF-DB dataset, we use our augmentation method to balance the long-tailed distribution. Our method can be used in not only FER tasks, but also more diverse data-hungry scenarios.

Introduction: The Long-Tail Problem

In the field of computer vision, datasets gathered "in the wild" often suffer from a long-tailed distribution. In the context of Facial Expression Recognition (FER), this means that while common emotions like "Happiness" or "Neutral" are well-represented, critical but rare emotions like "Fear" or "Disgust" have significantly fewer samples.

The long-tailed distribution of emotions in the RAF-DB dataset. Majority classes like 'Happiness' dominate, while 'Fear' and 'Disgust' form the sparse 'tail'.

This imbalance poses a major hurdle for AI models. Because the "tail" classes have so few examples, models fail to learn their nuances, leading to poor recognition accuracy for the very emotions that are often the most critical to detect in healthcare or safety monitoring. Traditional fixes like over-sampling (repeating rare images) often lead to over-fitting, while basic geometric augmentations (flipping or rotating) don't add enough semantic variety.

Methodology: Semantic Data Augmentation via VAE-GAN

The proposed solution is a Semantic Data Augmentation approach powered by a VAE-GAN (a hybrid of a Variational Auto-Encoder and a Generative Adversarial Network). This method allows for the generation of new, realistic samples by manipulating images in a compressed latent space.

The VAE-GAN architecture used for semantic augmentation. An encoder $E$ maps images to a latent space $z$, where they are augmented before being reconstructed by a generator $G$.

The process works in three main stages:

  1. Mapping to Latent Space: An encoder ($E$) takes a face image and compresses it into a "latent vector" ($z$). In this mathematical space, different directions correspond to semantic changes—like changing a person’s identity, pose, or lighting.
  2. Class-Conditional Augmentation: To create a new sample, the system calculates the covariance matrix ($\Sigma_c$) for all samples in a specific emotion class. This matrix identifies which directions in the latent space are "meaningful" for that emotion. By sampling new points ($z'$) based on this covariance, the system ensures the generated features remain realistic and semantically consistent.
  3. Image Reconstruction: A generator ($G$) takes the modified latent vector and turns it back into a full-resolution image. Because the VAE is paired with a Discriminator ($D$), the resulting images avoid the blurriness common in standard VAEs.

Experimental Evaluation

The study primarily utilizes the RAF-DB (Real-world Affective Faces Database), which contains 12,271 training images and 3,068 testing images. The dataset is highly imbalanced, with 4,772 "Happiness" samples compared to only 281 "Fear" samples.

The method was tested against two popular architectures: a standard ResNet34 and the state-of-the-art DAN (Distract Your Attention Network).

Model Augmentation Total Precision mAP (Mean Avg Precision)
ResNet34 None 0.7021 0.5587
ResNet34 Balanced (Resampling) 0.7322 0.6412
ResNet34 Ours (Semantic) 0.7443 0.6737
DAN None 0.8832 0.8198
DAN Ours (Semantic) 0.8924 0.8205

The results demonstrate that semantic augmentation consistently outperforms both the baseline and simple balanced resampling, particularly in mAP, which is a more robust metric for evaluating performance across imbalanced classes.

Limitations and Future Work

While effective, the authors note a few areas for growth:

  • Augmentation Rates: Future work could explore tailoring augmentation rates specifically to the rarity of a class (e.g., augmenting "Fear" more aggressively than "Sadness").
  • Complexity: Training a VAE-GAN is a computationally expensive prerequisite for the classification pipeline.
  • Generalization: While tested on FER, this method could potentially be applied to other "data-hungry" scenarios where long-tailed distributions exist.

Conclusion

Semantic data augmentation provides a powerful alternative to traditional geometric methods. By leveraging the latent space of a VAE-GAN and class-specific statistics, researchers can generate diverse, high-fidelity training data that helps models overcome the long-tail problem. This approach improves FER accuracy and holds promise for broader applications in computer vision.

Summary updated

Abstract

Facial Expression Recognition has a wide application prospect in social robotics, health care, driver fatigue monitoring, and many other practical scenarios. Automatic recognition of facial expressions has been extensively studied by the Computer Vision research society. But Facial Expression Recognition in real-world is still a challenging task, partially due to the long-tailed distribution of the dataset. Many recent studies use data augmentation for Long-Tailed Recognition tasks. In this paper, we propose a novel semantic augmentation method. By introducing randomness into the encoding of the source data in the latent space of VAE-GAN, new samples are generated. Then, for facial expression recognition in RAF-DB dataset, we use our augmentation method to balance the long-tailed distribution. Our method can be used in not only FER tasks, but also more diverse data-hungry scenarios.
BibTeX
@misc{li2024semantic,
  author = {Zijian Li and Yan Wang and Bowen Guan and Jiankai Yin},
  title = {{Semantic Data Augmentation for Long-tailed Facial Expression Recognition}},
  year = {2024},
  eprint = {2411.17254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.17254},
}
Figure