Abstract
Facial Expression Recognition (FER) in "in-the-wild" scenarios remains a significant challenge due to the scarcity of high-quality labeled data and the complex, multi-modal nature of human emotion. This paper presents ConCluGen, a novel multi-task self-supervised learning framework that leverages unlabeled video, audio, and text. By integrating contrastive alignment, semantic clustering, and generative reconstruction, ConCluGen learns robust representations that are competitive with or exceed several fully supervised state-of-the-art models.
Introduction
Human emotions are inherently multi-modal, conveyed through a combination of facial expressions, vocal prosody, and linguistic context. Traditional FER systems often struggle with "in-the-wild" data where labels are expensive to produce and visual cues alone may be ambiguous. ConCluGen addresses these gaps by utilizing self-supervised learning (SSL) to extract meaningful patterns from vast amounts of unlabeled multi-modal video data, such as celebrity interviews.
Methodology
The ConCluGen framework is built upon three core self-supervised objectives designed to capture both low-level features and high-level semantics across modalities.
- Multi-Modal Contrastive Learning (MMS): This task aligns representations of different modalities (video, audio, text) from the same temporal segment while distancing them from unrelated segments in the embedding space.
- Multi-Modal Clustering (Clu): To prevent over-discrimination, this loss groups semantically similar samples together, preserving the broader emotional structure of the data and mitigating "class collision."
- Multi-Modal Reconstruction (Gen): This generative task forces the model to reconstruct original features from latent embeddings, ensuring the preservation of essential low-level data characteristics.
The architecture utilizes fixed pre-trained backbones—ResNet for video, DAVENet for audio, and DistilBERT for text—focusing the learning process on the representation heads that fuse these signals.
Experimental Results
The model was pre-trained on the VoxCeleb2 dataset (145,000 unlabeled videos) and evaluated on benchmarks including CMU-MOSEI, MELD, and CAER.
As shown in the results, ConCluGen achieves a state-of-the-art accuracy of 66.48% on CMU-MOSEI, outperforming several fully supervised models that had access to labels during their entire training process. Key insights include:
- The Contextual Glue of Text: Text provides a significantly stronger semantic anchor for identifying complex emotions than audio when paired with video.
- Cross-Modal Synergy: Multi-modal pre-training enhances performance even when only a single modality (e.g., video) is used during downstream inference, suggesting that audio and text "teach" the visual encoder to identify more meaningful features.
- Robustness: Even with a frozen backbone (linear evaluation), the model achieved 60.0% accuracy on CMU-MOSEI, proving the robustness of the learned features.
Conclusion
ConCluGen demonstrates that multi-task self-supervised learning is a powerful alternative to supervised training for FER. By combining contrastive, clustering, and generative losses, the model captures the nuanced, multi-modal nature of human expression. Future work will explore end-to-end fine-tuning of encoders and the integration of additional modalities like facial landmarks or physiological signals.
Summary updated