Abstract
CLEF (Contrastive Learning with Text-Embedded framework) introduces a novel approach to facial behavior analysis by leveraging coarse-grained activity descriptions as semantic anchors. Traditional self-supervised methods often suffer from identity bias, where models learn to recognize individuals rather than facial movements. By using activity-based pairing and rich textual label descriptions, CLEF successfully disentangles facial expressions from subject identity. The framework achieves state-of-the-art performance in both Facial Action Unit (AU) recognition and emotion classification across six major benchmarks, demonstrating significant improvements in zero-shot generalization.
Introduction: The Challenge of Identity Bias
Self-supervised contrastive learning typically functions by pulling an image (an "anchor") closer to its augmented versions while pushing it away from other images in a batch. In the context of facial analysis, this often leads to identity bias. Because datasets frequently contain many images of the same individuals, a model might inadvertently push away two different people performing the same expression, leading it to prioritize identity over facial behavior. Solving this usually requires expensive, fine-grained annotations. CLEF addresses this by utilizing "coarse" activity labels (e.g., "Subject is watching a sad movie") already present in many datasets to provide a more effective training signal.
Methodology
The CLEF framework operates in two distinct stages, building upon the CLIP (Contrastive Language-Image Pre-training) architecture.
Stage 1: Weakly-Supervised Pre-training
Instead of standard self-supervision, CLEF employs activity-based pairing. Images captured during the same activity—such as "listening to a joke"—are treated as positive pairs, regardless of whether they represent different frames or different individuals. By contrasting these images against the textual description of the activity, the image encoder is forced to focus on shared semantic features (the expression) rather than individual identity features that remain constant across different activities.
Stage 2: Text-Driven Fine-tuning
In the second stage, the model transitions to recognizing specific labels like "Happiness" or "AU6 - Cheek Raiser." Rather than relying solely on label names, CLEF utilizes detailed label descriptions. For instance, the label "Happiness" is supplemented with descriptions such as "the eyes squint slightly, wrinkles appear at the corners..." This provides a rich semantic bridge, allowing the model to leverage pre-trained linguistic knowledge to understand the physical appearance of a behavior before seeing specific training examples.
Key Results and Insights
CLEF was evaluated on six major benchmarks, including BP4D, DISFA, and RAF-DB.
- State-of-the-Art Performance: The model achieved the highest average F1-scores on BP4D (65.9) and DISFA (64.8) for AU recognition. It also set new standards for emotion recognition on RAF-DB (90.09% accuracy) and FERPlus (89.74%).
- Zero-Shot Superiority: CLEF demonstrated remarkable generalization. In zero-shot tests on RAF-DB, it achieved 29.14% accuracy, significantly outperforming the massive FaRL model (16.21%). This suggests that the text-driven approach allows the model to understand unseen behaviors through their semantic descriptions.
- Ablation Studies: Removing either the activity-based pre-training or the detailed label descriptions resulted in a measurable performance drop, confirming that both coarse activity text and fine-grained label text are essential components of the framework.
Conclusion
CLEF demonstrates that coarse-grained activity descriptions, often overlooked in facial research, can serve as powerful identity-neutral supervisors. By bridging the gap between numerical labels and descriptive text, the framework achieves superior disentanglement and generalization. Future work will focus on automating the generation of these descriptions to extend the method to large-scale "in-the-wild" datasets that currently lack such metadata.
Summary updated