Learning to Discover Generalized Facial Expressions

Tingzhang Luo, Yichao Liu, Yuanyuan Liu, Andi Zhang, Xin Wang, Chang Tang, Zhe ChenarXiv 2024 · DOI · arXiv

Abstract

Generalized Facial Expression Category Discovery (G-FACE) is a challenging task that requires models to recognize known emotions while simultaneously discovering new, complex expression categories in unlabeled data. Traditional methods often struggle with the subtle visual differences and distributional shifts inherent in facial data. This paper introduces DIG-FACE (De-biased G-FACE), a framework that mathematically bounds implicit distributional biases and employs a hierarchical refinement strategy to mitigate explicit visual overlaps. Experimental results on major benchmarks like RAF-DB and AffectNet demonstrate that DIG-FACE significantly outperforms existing state-of-the-art category discovery methods.

The Challenge of Open-World Emotion Recognition

Most facial expression recognition (FER) systems operate in a "closed-set" environment, trained on a fixed list of basic emotions such as happiness, sadness, or surprise. However, real-world human emotion is nuanced and often involves composite expressions (e.g., "happily surprised") that do not fit into predefined categories. Manually labeling every possible complex expression is prohibitively expensive, necessitating Generalized Category Discovery (GCD).

Standard GCD methods designed for distinct objects (like cars vs. dogs) often fail on facial data due to two primary biases:

  1. Implicit Bias: A distributional gap exists between known and new categories in unlabeled data, which shifts decision boundaries during training.
  2. Explicit Bias: Facial expressions are visually subtle. The difference between "Anger" and "Disgust" is far smaller than the difference between a "Cat" and a "Bus," leading to heavily overlapped feature spaces.

The DIG-FACE Framework

To address these challenges, DIG-FACE employs a two-pronged debiasing strategy designed to refine the model's understanding of both global distributions and fine-grained facial features.

The DIG-FACE framework architecture, featuring Implicit Debiasing via F-discrepancy estimation and Explicit Debiasing through hierarchical refinement.

Implicit Debiasing

The researchers treat the distributional gap as a mathematical optimization problem. They define an F-discrepancy metric to estimate the upper bound of the bias introduced by new categories. By using an adversarial "Min-Max" training process with an auxiliary classification head, the model is forced to minimize this discrepancy, ensuring robust decision boundaries even as new classes are introduced.

Explicit Debiasing

To resolve the problem of overlapping features in subtle expressions, DIG-FACE uses a three-level hierarchical refinement:

  • Sample-level: A "Confusing Sample Mining" strategy dynamically increases the weight of hard-to-distinguish samples during training.
  • Triplet-level: Contrastive learning—both supervised for known classes and self-supervised for unlabeled data—ensures the model captures fine-grained textures like specific eye or mouth movements.
  • Distribution-level: A supervised clustering loss encourages "intra-class compactness" (keeping similar emotions together) and "inter-class separation" (pushing different emotions apart).

Experimental Results

DIG-FACE was evaluated on several major FER benchmarks, including RAF-DB (Basic & Compound), FERPlus, and AffectNet. The model set a new state-of-the-art standard across these datasets.

On the RAF-DB dataset, DIG-FACE improved the accuracy of recognizing known ("Old") classes by 16.2% compared to the previous best method, SimGCD. When tasked with discovering complex compound expressions, it achieved a 20.8% improvement in old category accuracy and a 4.9% boost in new category discovery.

t-SNE visualizations comparing a baseline model to DIG-FACE, showing significantly clearer separation and clustering of both old and new facial expression categories.

Visualizations of the model's attention maps reveal that DIG-FACE focuses more intensely on key communicative areas—the cheeks, eyes, and mouth corners—than previous baselines. Furthermore, the t-SNE plots above demonstrate that DIG-FACE effectively untangles the overlapped feature spaces that plague standard GCD methods, resulting in distinct, well-separated clusters for both known and discovered expressions.

Conclusion

DIG-FACE demonstrates that by quantifying distributional bias and applying hierarchical feature refinement, models can successfully navigate the "fine-grained" challenges of facial expression discovery. While the current framework assumes a known number of potential clusters, it provides a robust foundation for future "open-world" systems that can incrementally learn and categorize the full spectrum of human emotion without exhaustive manual supervision.

Summary updated

Abstract

We introduce a novel task, Generalized Facial Expression Category Discovery (G-FACE), that discovers new, unseen facial expressions while recognizing known categories effectively. Even though there are generalized category discovery methods for natural images, they show compromised performance on G-FACE. We identified two biases that affect the learning: implicit bias, coming from an underlying distributional gap between new categories in unlabeled data and known categories in labeled data, and explicit bias, coming from shifted preference on explicit visual facial change characteristics from known expressions to unknown expressions. By addressing the challenges caused by both biases, we propose a Debiased G-FACE method, namely DIG-FACE, that facilitates the debiasing of both implicit and explicit biases. In the implicit debiasing process of DIG-FACE, we devise a novel learning strategy that aims at estimating and minimizing the upper bound of implicit bias. In the explicit debiasing process, we optimize the model's ability to handle nuanced visual facial expression data by introducing a hierarchical category-discrimination refinement strategy: sample-level, triplet-level, and distribution-level optimizations. Extensive experiments demonstrate that our DIG-FACE significantly enhances recognition accuracy for both known and new categories, setting a first-of-its-kind standard for the task.
BibTeX
@misc{luo2024learning,
  author = {Tingzhang Luo and Yichao Liu and Yuanyuan Liu and Andi Zhang and Xin Wang and Chang Tang and Zhe Chen},
  title = {{Learning to Discover Generalized Facial Expressions}},
  year = {2024},
  eprint = {2409.20098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.20098},
}
Figure