Abstract
Facial Expression Recognition (FER) models often struggle with domain shift when deployed in real-world environments. While Unsupervised Domain Adaptation (UDA) can mitigate this, privacy concerns often prevent access to original source datasets. This paper presents CluP (Cluster-level Pseudo-labelling), the first framework for Source-Free Unsupervised Domain Adaptation (SFUDA) in FER. CluP utilizes self-supervised pretraining and a novel cluster purity metric to generate reliable pseudo-labels from unlabeled target data. Experimental results show that CluP outperforms existing SFUDA methods and achieves performance comparable to traditional UDA techniques.
Problem Statement: Domain Shift and Privacy
Facial Expression Recognition (FER) models frequently fail in production due to domain shift—the discrepancy between training (source) images and real-world (target) images. While Unsupervised Domain Adaptation (UDA) is the standard solution, it requires simultaneous access to both source and target data. In FER, this is often impossible due to privacy regulations like GDPR, as face images are highly sensitive.
This leads to the Source-Free (SFUDA) setting, where adaptation must occur using only a pretrained model and a new, unlabeled target dataset. The primary challenge is that the source model’s confidence is often unreliable on target data, causing standard pseudo-labelling techniques to fail.
The CluP Framework
The authors propose CluP (Cluster-level Pseudo-labelling), a three-stage framework designed to adapt models without source data access.
Methodology
- Self-supervised Pretraining (SSL): Instead of initializing the target model with potentially biased source weights, the feature extractor is "warmed up" on target data using SSL (e.g., SwAV). This allows the model to learn the underlying structure of the target images without labels.
- Cluster-level Pseudo-labelling: Rather than trusting individual predictions, CluP groups target features using K-means. It assigns a "majority vote" label to each cluster, identifying where the source model is consistent.
- Final Training: The SSL-warmed backbone is combined with a new classifier and trained using the high-confidence pseudo-labels generated in the previous stage.
Cluster Purity Filtering
A critical component of CluP is the Cluster Purity ($s_j$) metric. It calculates the percentage of samples in a cluster that agree with the majority label. To ensure high-quality pseudo-labels, only clusters exceeding a specific purity threshold (based on the $Q$-th percentile of their category) are retained for training. This "purity over confidence" approach identifies regions of the feature space where the model's predictions are stable and likely correct.
Experimental Evaluation
The study evaluates CluP across four major FER benchmarks: AFE (Asian film images), RAF-DB (real-world internet images), ExpW (diverse in-the-wild images), and FER2013 (low-resolution grayscale images).
Key Results
- State-of-the-Art in SFUDA: CluP significantly outperforms existing SFUDA methods like SHOT. In the AFE $\rightarrow$ ExpW task, CluP achieved 65.00% accuracy, compared to SHOT's 54.12%.
- Closing the Gap with UDA: Remarkably, CluP performs on par with—and sometimes better than—traditional UDA methods that require source data. For example, on AFE $\rightarrow$ FER2013, CluP reached 52.51%, surpassing the UDA method AGRA (51.95%).
- Robustness: The Cluster Purity metric consistently outperformed standard confidence-based filtering across all tested thresholds.
Key Insights
- Source Weights Can Be a Hindrance: Initializing a target model with source weights can limit adaptability. SSL-based pretraining allows the model to better fit the target domain's feature space.
- Over-clustering for Balance: To handle the inherent class imbalance in FER datasets (e.g., many "Happy" faces, few "Disgust" faces), the authors use "over-clustering" ($K \gg N$ classes) to ensure minority classes are captured in distinct clusters.
Conclusion and Future Work
CluP demonstrates that effective domain adaptation is possible without compromising data privacy. While it sets a new benchmark for SFUDA in FER, challenges remain in handling extreme minority classes like "Disgust." Future work will focus on moving toward online SFUDA for continuous adaptation and reducing the computational cost of the multi-stage training process.
Summary updated