Abstract
Facial emotion recognition (FER) in static images often lacks the temporal context inherent in human expressions. This paper proposes a self-supervised contrastive pre-training framework that leverages video sequences to learn richer emotion embeddings. By employing triplet loss with specific sampling strategies—focusing on emotion discrimination and intensity refinement—the model captures the dynamic evolution of expressions. Experimental results on the FER-2013 dataset demonstrate that this approach outperforms standard supervised baselines, achieving a 69.15% accuracy and surpassing previous competition leaders.
Introduction
Recognizing emotions from a single facial image is a challenging task due to high inter-individual variability and the difficulty of obtaining large-scale, accurately labeled datasets. Traditional state-of-the-art models often rely on complex ensembles or hand-designed features, yet they frequently overlook the fact that emotions are dynamic processes evolving from a neutral state to a peak intensity. This research addresses this gap by "borrowing" temporal information from video data to enhance single-image classification, arguing that the transitions captured in video provide essential signals for training robust classifiers.
Methodology
The core methodology involves Contrastive Pre-training using a triplet loss function. Instead of processing images in isolation, the model evaluates three samples simultaneously: an anchor, a positive (similar), and a negative (dissimilar). The objective is to minimize the distance between the anchor and positive samples in a high-dimensional embedding space while maximizing the distance to the negative sample.
The authors propose two specific sampling methods to generate these triplets from video data:
- Emotion Discrimination (Method 1): The anchor and positive are frames of the same emotion at different intensities, while the negative is a frame representing a different emotion from the same subject. This teaches the model to distinguish between emotion categories while remaining invariant to individual facial features.
- Intensity Refinement (Method 2): All three frames are drawn from the same video sequence. The anchor is the lowest intensity (neutral), the positive is medium intensity, and the negative is the highest intensity (peak). This forces the model to understand the progression and "spectrum" of a single emotion.
The study utilized the Cohn-Kanade (CK+) dataset for pre-training, which provides controlled sequences of expressions moving from neutral to peak. Evaluation was performed on the FER-2013 benchmark, a standard for emotion classification tasks.
Results and Discussion
The proposed "Contrastive Transformer" (CT512), utilizing both sampling methods, achieved an accuracy of 69.15% on FER-2013. This result represents a significant improvement over the standard InceptionV1 baseline (67.73%) and outperforms the top-performing "MAXIM MILAKOV" entry (68.82%) from the original Kaggle competition.
Key insights from the experiments include:
- Intensity as Self-Supervision: Treating different stages of an expression as related signals allows the model to learn robust representations without requiring manual labels for every frame. The inherent temporal structure of video provides a powerful self-supervisory signal.
- Sequential Training Efficacy: The authors found that the order of training is critical. Using Method 1 first allows the model to learn broad categorical distinctions (e.g., Happy vs. Sad), while following up with Method 2 helps the model cluster those categories more tightly across varying intensities.
Conclusion
This work demonstrates the efficacy of self-supervised video-to-image transfer for emotion recognition. By leveraging the dynamic nature of expressions captured in video, models can learn more nuanced features for static image classification. Future research will explore "hard triplet mining" to improve sampling efficiency and test the approach on more diverse, "in-the-wild" datasets and modern architectures like Vision Transformers (ViTs).
Summary updated