Analysis of Semi-Supervised Methods for Facial Expression Recognition

Shuvendu Roy, Ali Etemad

Abstract

Facial Expression Recognition (FER) typically requires large-scale, human-annotated datasets, which are costly and time-consuming to produce. This paper evaluates the efficacy of semi-supervised learning (SSL) in reducing this dependency. By benchmarking eight state-of-the-art SSL methods across three diverse datasets, the authors demonstrate that models trained with as few as 250 labeled samples per class can achieve performance nearly equivalent to fully supervised models. FixMatch is identified as the most robust method for FER tasks, offering a significant reduction in annotation effort without sacrificing accuracy.

1. Introduction

Deep learning has revolutionized Facial Expression Recognition (FER), yet these models remain notoriously data-hungry. Achieving high accuracy typically requires tens of thousands of images meticulously labeled by humans—a process that is expensive, time-consuming, and often subjective. While semi-supervised learning (SSL) has seen massive success in general object recognition by combining small amounts of labeled data with large pools of unlabeled data, its application to the specific nuances of human emotion has remained largely unexplored.

Overview of semi-supervised learning for FER.

This paper provides the first comprehensive benchmark of eight state-of-the-art SSL methods for FER, evaluating how well general-purpose algorithms adapt to emotional data and establishing a baseline for future research in the field.

2. Methodology

The authors benchmarked eight prominent SSL algorithms, which generally fall into three categories:

  1. Pseudo-labeling: The model predicts labels for unlabeled data; those with high confidence are treated as "ground truth" for further training.
  2. Consistency Regularization: The model is forced to produce the same output even if the input image is slightly perturbed (e.g., flipped or rotated).
  3. Hybrid Methods: These combine both approaches. A standout example is FixMatch, which uses a "weakly" augmented image to generate a pseudo-label and then trains the model to predict that same label on a "strongly" augmented version of the same image.

To ensure performance gains were attributable to the SSL techniques rather than architectural advantages, all methods utilized a standard ResNet-18 backbone.

3. Experimental Setup

The methods were tested across three diverse FER benchmarks to ensure robustness:

  • FER13: 35k low-resolution (48x48) images from the Google Image search API.
  • RAF-DB: An "in-the-wild" dataset containing 15k images annotated by over 300 human observers.
  • AffectNet: A massive-scale dataset with over 284k images, representing one of the most challenging benchmarks in the field.

4. Results and Discussion

The quantitative results highlight massive efficiency gains. For all three datasets, using just 250 labeled samples per class yielded results within 2-4% of the accuracy achieved using the entire labeled dataset. On AffectNet, the best SSL method achieved 51.25% accuracy using 140 times less labeled data than the full supervised version (54.91%).

Confusion matrices for the best semi-supervised model on each dataset.

Key Insights

  • FixMatch Dominance: FixMatch achieved the best results in 8 out of 12 experimental settings, benefiting from the synergy between high-confidence pseudo-labeling and diverse data augmentations.
  • SSL vs. Supervised: When limited to the same small amount of labeled data, SSL methods consistently outperformed purely supervised training. On AffectNet, the best SSL method was 11% more accurate than its supervised counterpart using the same label count.
  • Hyperparameter Sensitivity: Parameters optimized for general object recognition (like CIFAR-10) do not always translate to FER. For instance, the "Pseudo-label" method required a higher weight for unlabeled loss ($\lambda$) than standard defaults to perform optimally.

5. Conclusion

This study proves that general SSL methods work remarkably well for FER, significantly reducing the cost of human annotation. While "vanilla" versions of these algorithms are effective, future work could involve developing FER-specific augmentations or architectural tweaks to address the inherent subjectivity of human emotions, such as the common confusion between "sad" and "neutral" expressions. Additionally, exploring how these methods handle class imbalance remains a critical open question for real-world applications.

Summary updated

BibTeX
@inproceedings{roy2022analysis,
  author = {Shuvendu Roy and Ali Etemad},
  title = {{Analysis of Semi-Supervised Methods for Facial Expression Recognition}},
  booktitle = {ACII},
  year = {2022},
  doi = {10.1109/acii55700.2022.9953876},
}

Loading related content...

Figure