Contrastive Adversarial Learning for Person Independent Facial Emotion Recognition

Dae Ha Kim, Byung Cheol SongAAAI 2021 · DOI

Introduction

Facial Emotion Recognition (FER) systems often struggle with "person-dependency," where a model inadvertently learns an individual's unique facial structure or intrinsic mood rather than the universal signals of emotion. This challenge is particularly acute in the Arousal-Valence (AV) domain, where emotions are mapped onto a continuous scale (e.g., active vs. passive, positive vs. negative). While intense expressions are easily identified, subtle or "weak" emotions are frequently masked by personal facial characteristics.

To address this, the authors introduce the Contrastive Adversarial Framework (CAF). This approach leverages adversarial learning to bridge the gap between weak and strong emotional expressions, employing a novel adaptive contrastive loss to achieve state-of-the-art results in continuous emotion estimation.

Methodology

The CAF operates through a three-stage pipeline designed to extract the "emotional essence" of a face, independent of the subject's identity.

1. Emotional Binarization

The process begins with Otsu thresholding, a classic image processing technique repurposed here to find optimal decision boundaries in the AV space. This allows the researchers to categorize the dataset into two distinct groups: "strong" and "weak" emotions based on their labeled arousal and valence values.

2. Adversarial Learning in Latent Space

The core of the framework is a game between an Encoder ($f_e$) and a Critic ($f_c$) network. The Encoder is tasked with mapping weak emotion samples into a latent space such that they become indistinguishable from strong emotion samples to the Critic. By forcing this alignment, the network learns to ignore the "noise" of a person's resting face and focus on the underlying emotional features that characterize both subtle and intense expressions.

The Contrastive Adversarial Framework (CAF) architecture, illustrating the flow from image grouping via Otsu thresholding to adversarial learning in the latent feature space.

3. Contrastive Loss with Adaptive Margin

Unlike standard adversarial setups that rely on global statistical distances (like Maximum Mean Discrepancy), CAF utilizes a contrastive loss function. This allows the model to capture local distributions and relationships between individual samples. A key innovation is the adaptive margin ($\hat{\alpha}$), which adjusts based on Mutual Information (MI). This ensures the adversarial task remains appropriately challenging throughout the training process, preventing overfitting and ensuring the model continues to learn useful features.

Experimental Setup and Datasets

The model was rigorously evaluated on three major benchmarks for continuous emotion recognition:

  • AffectNet: A massive dataset containing over 440,000 images annotated with AV labels.
  • AFEW-VA: Derived from movie clips, providing per-frame AV labels in dynamic environments.
  • Aff-Wild: Used to test the model's generalization capabilities across diverse video-based reactions.

Key Results

The proposed CAF achieved State-of-the-Art (SOTA) performance across multiple metrics, demonstrating the effectiveness of the weak-to-strong mapping strategy.

On the AffectNet dataset, using a ResNet18 backbone, the model achieved a Root Mean Squared Error (RMSE) of 0.2186 for Valence and 0.1873 for Arousal. These results significantly outperform previous benchmarks, including complex models like BreG-NeXt. Furthermore, despite being a static model that processes single frames, CAF outperformed several temporal models on the AFEW-VA dataset in terms of Valence RMSE and Pearson Correlation Coefficient (PCC).

Performance comparison on the AffectNet dataset, showing the proposed method (Ours) achieving state-of-the-art results in RMSE, PCC, and CCC compared to existing backbones.

Qualitative analysis via T-SNE visualizations confirmed that the model effectively clustered emotions by intensity and type, proving that the discriminative learning successfully created a person-independent feature space.

Discussion and Future Work

The success of CAF highlights the potential of treating weak expressions as "noisy" versions of strong ones. By aligning these distributions, the model effectively filters out identity-specific information.

However, limitations remain. The current framework processes images frame-by-frame; the authors suggest that integrating temporal information—how expressions evolve over time—could further enhance performance on video datasets. Additionally, while the adaptive margin calculation adds training complexity, the inference remains efficient. Future research may explore applying this contrastive adversarial approach to other deep metric learning domains beyond facial recognition.

Summary updated

BibTeX
@inproceedings{kim2021contrastive,
  author = {Dae Ha Kim and Byung Cheol Song},
  title = {{Contrastive Adversarial Learning for Person Independent Facial Emotion Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i7.16743},
}
Figure