Towards Regularized Mixture of Predictions for Class-Imbalanced Semi-Supervised Facial Expression Recognition

Hangyu Li, Yixin Zhang, Jiangchao Yao, Nannan Wang, Bo HanIJCAI 2025 · DOI

Abstract

Semi-supervised facial expression recognition (SSFER) leverages small labeled datasets alongside large unlabeled pools to train robust models. However, real-world emotion datasets are often highly imbalanced, leading models to over-predict majority classes like "Happiness" and "Neutral." We propose Regularized Mixture of Predictions (ReMoP), a framework that mitigates this bias by integrating feature-level similarity into the pseudo-labeling process and enforcing a Simplex Equiangular Tight Frame (ETF) structure on class centers. Our approach ensures that minority classes remain discriminative in the feature space. Experimental results on RAF-DB, FERPlus, and CK+ demonstrate that ReMoP consistently outperforms state-of-the-art methods, particularly in mean accuracy across imbalanced classes.

Introduction

Facial expression recognition (FER) is a critical component of human-computer interaction, yet its performance is often hampered by the scarcity of labeled data. Semi-supervised learning (SSL) offers a solution by utilizing unlabeled data through pseudo-labeling. However, a significant challenge remains: class imbalance. In datasets like RAF-DB, majority classes can outnumber minority classes (e.g., "Fear" or "Disgust") by a factor of 100 or more.

Standard SSL methods tend to produce biased pseudo-labels, favoring majority classes and creating a feedback loop that marginalizes minority emotions. While existing solutions often adjust model outputs at the logit level, they frequently neglect the underlying feature geometry. In FER, high intra-class variation makes maintaining a well-structured feature space essential for distinguishing between subtle emotional cues.

Methodology

The ReMoP framework introduces two synergistic components to address these challenges: the Mixture of Predictions and Class Regularization.

Mixture of Predictions

Instead of relying solely on a linear classifier, which is prone to majority-class bias, ReMoP incorporates a "second opinion" based on feature similarity. The model maintains a set of class centers in the feature space. For an unlabeled sample, it calculates cosine similarity to these centers to generate a feature-based prediction. The final pseudo-label is a weighted mixture ($\beta$) of the linear classifier's output and this feature similarity score. This hybrid approach provides a more stable prediction for minority classes.

Class Regularization ($L_{reg}$)

To prevent minority classes from being "crowded out," ReMoP enforces a rigid geometric structure on the feature space. Drawing on the concept of Neural Collapse (NC), the authors introduce a regularization term that forces class centers to form a Simplex Equiangular Tight Frame (ETF). This mathematical constraint ensures that class centers are maximally separated and equidistant, preserving the discriminative power of the model even for classes with very few samples.

The overall pipeline of ReMoP, illustrating the mixture of predictions for pseudo-labeling and the center update mechanism for class regularization.

Experimental Setup

We evaluated ReMoP on four major benchmarks:

  • RAF-DB: Real-world dataset with 7 basic emotions.
  • FERPlus: An 8-class extension of FER2013.
  • CK+: A lab-controlled dataset for fine-grained recognition.
  • AffectNet: Used to provide "in-the-wild" unlabeled samples for robustness testing.

The method was integrated into standard SSL pipelines like FixMatch and Ada-CM to demonstrate its "plug-and-play" capability.

Results and Discussion

ReMoP consistently outperformed state-of-the-art class-imbalanced SSL methods such as ABC and CPE. On the RAF-DB dataset with a high imbalance ratio ($\gamma=150$), ReMoP achieved an overall accuracy of 81.42%, a 3% improvement over the competitive CPE baseline.

The most significant gains were observed in mean accuracy, which averages performance across all classes. On FERPlus, ReMoP improved mean accuracy by 3-4% over standard FixMatch, indicating much better recognition of minority emotions.

Qualitative Analysis

The impact of ReMoP is clearly visible in feature space visualizations. Without ReMoP, minority class features often bleed into majority clusters. With ReMoP, the features form tighter, more distinct clusters, reflecting the effectiveness of the ETF regularization.

t-SNE visualizations of feature distributions on RAF-DB comparing the baseline with ReMoP.

Conclusion

ReMoP provides a robust framework for handling class imbalance in SSFER by combining feature-level similarity with geometric constraints. By forcing the model to maintain a Simplex ETF structure, it prevents the marginalization of minority emotions. Future work includes developing adaptive mechanisms for hyperparameter tuning and exploring class discovery in open-world scenarios where unlabeled data may contain novel expressions.

Summary updated

BibTeX
@inproceedings{li2025towards,
  author = {Hangyu Li and Yixin Zhang and Jiangchao Yao and Nannan Wang and Bo Han},
  title = {{Towards Regularized Mixture of Predictions for Class-Imbalanced Semi-Supervised Facial Expression Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/154},
}
Figure