FAIR-FER: A Latent Alignment Approach for Mitigating Bias in Facial Expression Recognition (Student Abstract)

Syed Sameen Ahmad Rizvi, Aryan Seth, Pratik NarangAAAI 2024 · DOI

TL;DR: FAIR-FER introduces a novel approach to reducing demographic bias in Facial Expression Recognition (FER) by aligning the latent spaces of different groups. By using a shared-weight Variational Autoencoder (VAE) and an adversarial discriminator, the model learns to extract emotion-related features while intentionally ignoring protected attributes like age, gender, and race.

Introduction

Facial Expression Recognition (FER) systems are increasingly integrated into various technologies, yet they often suffer from demographic biases inherited from human-annotated datasets. For instance, studies have shown that human labelers frequently associate younger faces with positive emotions (like happiness) and older faces with negative ones (like sadness or disgust). When deep learning models train on such data, they learn "shortcuts"—predicting emotions based on demographic features rather than actual facial muscle movements. This leads to a lack of "equal-odds" fairness, where accuracy varies significantly across different groups. FAIR-FER addresses this by ensuring the model's internal representation of a face is "blind" to protected attributes like age, gender, and race.

Methodology: Attribute Disentanglement

The authors propose a two-part architecture designed for attribute disentanglement. The core idea is to align the latent representations of different demographic groups so that they are indistinguishable to downstream classifiers.

FAIR-FER Architecture: The model uses an encoder to map images to a latent space, which is then processed by MBConv blocks for classification.

The system consists of several key components:

  1. Shared-Weight VAE: A Variational Autoencoder encodes images into a latent space. The encoder's weights are shared across all demographic groups to promote a unified representation. A generator is used during training to ensure the latent space retains structural facial information through reconstruction loss.
  2. Adversarial Alignment: An adversarial discriminator attempts to predict protected attributes (e.g., age, race) from the latent code. The encoder is trained to "fool" this discriminator, effectively scrubbing demographic-specific information from the representation. The training objective balances KL-Divergence, discriminator loss, and style-reconstruction loss.
  3. MBConv Classifier: The resulting "fair" latent representation is fed into a classification module using Mobile Inverted Bottleneck Convolution (MBConv) blocks to predict one of seven basic emotions.

Experimental Results

The model was evaluated on the RAF-DB (Real-world Affective Faces Database), which contains 12,271 training images and 3,068 test images annotated with seven basic emotions and demographic metadata. The results demonstrate that FAIR-FER not only improves fairness but also enhances overall predictive accuracy.

Comparison of bias mitigation metrics across protected attributes. Higher values indicate better fairness (equal odds).

As shown in the results, FAIR-FER achieved an overall accuracy of 76.1%, surpassing the baseline of 74.8%. More importantly, it significantly improved fairness metrics for Race (from 91.6 to 94.2) and Age (from 82.1 to 84.8). By forcing the model to ignore demographic "noise," it is compelled to focus more effectively on the actual visual cues relevant to facial expressions, such as muscle movements. This effectively counteracts the "happy adolescent" bias where models over-predict positive emotions for younger faces.

Conclusion and Future Work

FAIR-FER provides a robust proof-of-concept for using latent alignment to shield FER models from societal biases. While currently tested on basic emotions, future work will explore its scalability to complex "in-the-wild" datasets and its generalizability to other domains like medical imaging. Although the adversarial VAE pipeline introduces additional computational overhead, the gains in both fairness and accuracy make it a promising direction for ethical AI development.

Summary updated

BibTeX
@inproceedings{rizvi2024fair,
  author = {Syed Sameen Ahmad Rizvi and Aryan Seth and Pratik Narang},
  title = {{FAIR-FER: A Latent Alignment Approach for Mitigating Bias in Facial Expression Recognition (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30503},
}
Figure