Emotion Separation and Recognition From a Facial Expression by Generating the Poker Face With Vision Transformers

Jia Li, Jiantao Nie, Dan Guo, Richang Hong, Meng Wang

TL;DR

This paper introduces PF-ViT, a novel approach for facial expression recognition (FER) that uses Vision Transformers (ViTs) and a Generative Adversarial Network (GAN) to disentangle emotional content from facial images. By generating a "poker face" (emotionless version) of the input, PF-ViT achieves state-of-the-art accuracy on challenging in-the-wild FER datasets, even without paired emotion-identity data.

What Problem Does This Paper Address?

Facial Expression Recognition (FER) is crucial for many applications, but accurately identifying emotions from images, especially in real-world scenarios ("in-the-wild"), is difficult. This is due to several factors:

  • Ambiguous Emotion Labels: Datasets often have noisy or unclear labels.
  • Inherent Ambiguity: Facial expressions themselves can be subtle and hard to distinguish.
  • Irrelevant Factors: Factors like identity, head pose, and illumination can interfere with emotion recognition.
  • Data Scarcity: Training robust models often requires large, well-annotated datasets, which are expensive to create, especially for diverse real-world conditions.
  • Lack of Fine-Grained Supervision: Directly mapping an image to an emotion label doesn't explicitly guide the model to focus on the specific facial features that convey emotion.

The authors aim to overcome these challenges by disentangling the emotional component of a facial expression from all other irrelevant information, allowing for more robust and accurate recognition.

Methods

The proposed method, PF-ViT (Poker Face Vision Transformer), is built upon two key ideas: Masked Autoencoders (MAE) for robust representation learning and a Generative Adversarial Network (GAN) for feature disentanglement.

  1. MAE Pre-training:
    • Vanilla Vision Transformers (ViTs) are used as the backbone.
  2. These ViTs are first pre-trained using an MAE framework on a large, unlabeled facial expression dataset (AffectNet-270K). MAE involves masking out a significant portion of image patches and training the model to reconstruct them. This self-supervised approach helps ViTs learn powerful, generalizable facial representations without relying on emotion labels.
  3. GAN-based Disentanglement (PF-MAG):
    • The pre-trained ViT encoder is then integrated into a GAN framework (PF-MAG).
  4. Token Separator (S): A novel component that takes the encoded facial representation and decomposes it into two orthogonal components:
  5. ve: The pure emotional component.
  6. vp: The emotion-irrelevant component (preserving identity, pose, etc.).
  7. Image Generator (G): This generator takes vp (and mask tokens) to synthesize a realistic "poker face" (emotionless version of the input). It also reconstructs the original face using both ve and vp.
  8. Discriminator (D): This discriminator tries to distinguish between real and generated faces, and also acts as a multi-label classifier for emotions.
  9. Adversarial Training: The GAN is trained adversarially. The generator tries to fool the discriminator by creating realistic poker faces, while the discriminator tries to identify fakes and correctly classify emotions. This process forces the token separator to effectively disentangle ve and vp.
  10. FER Classification Head (C): A linear classifier is attached to the emotional component ve to perform the final FER task.

During inference, the generator and discriminator can be removed, leaving only the encoder, separator, and classifier for efficient FER.

Datasets

  • Training (MAE Pre-training): AffectNet-270K (a large subset of AffectNet, manually labeled, ~269,316 images).
  • Training/Fine-tuning and Evaluation:
    • RAF-DB: A widely used in-the-wild FER dataset (29,672 images, 7 basic expressions).
  • AffectNet-7 / AffectNet-8: Large real-world FER datasets (AffectNet-7: 7 basic expressions; AffectNet-8: 7 basic expressions + contempt).
  • FERPlus: An extended version of FER2013 (8 emotion classes).

The paper emphasizes that their method does not require paired images of the same person with different expressions for training the GAN component.

Key Insights and Ideas

  • ViTs + MAE = Powerful Representations: Vanilla ViTs, when pre-trained with MAE on unlabeled facial data, can achieve surprisingly strong FER performance, even outperforming models pre-trained on general image datasets like ImageNet. This highlights the effectiveness of self-supervised learning for learning domain-specific features.
  • Poker Face Generation as Disentanglement Aid: The auxiliary task of generating a realistic poker face is crucial. It acts as a strong supervisory signal, forcing the model to separate emotional cues (ve) from all other facial details (vp). This is more effective than traditional methods that might only try to remove identity or pose.
  • Orthogonality is Key: Enforcing orthogonality between the emotional (ve) and emotion-irrelevant (vp) components is vital for effective disentanglement. Using a pre-trained face recognition model (Incept-ResV1) to guide this orthogonality proved more effective than simpler methods.
  • All Patches Matter: Unlike standard MAE, PF-ViT utilizes all image patches (not just a subset) in its generator for poker face synthesis. This is crucial for generating high-quality, realistic emotionless faces and achieving better FER performance.
  • Vanilla ViTs Can Compete: The paper demonstrates that pure ViT architectures, when properly pre-trained and fine-tuned, can achieve state-of-the-art results in FER, challenging the notion that hybrid CNN-Transformer models are always superior.

Key Results

  • State-of-the-Art Accuracy: PF-ViT achieved the highest reported accuracies on four popular in-the-wild FER datasets:
  • RAF-DB: 92.07%
    • AffectNet-7: 67.23%
    • AffectNet-8: 64.10%
    • FERPlus: 91.16%
  • Improved Performance with MAE: MAE pre-training consistently improved ViT performance by significant margins (e.g., +3.35% on RAF-DB) compared to standard supervised pre-training.
  • Efficiency: While the full PF-MAG framework includes a generator and discriminator for training, the final PF-ViT model for inference is efficient, with only a small increase in parameters and FLOPs compared to the MAE-ViT baseline.
  • Qualitative Results: Visualizations show that PF-ViT successfully separates emotional expressions, generating realistic poker faces that retain identity and other non-emotional features. Attention maps indicate PF-ViT focuses more on global, emotion-relevant features.

Limitations and Future Work

  • Ambiguous Expressions: The model still struggles with inherently ambiguous facial expressions that are difficult even for human annotators to classify.
  • FID Score: While the generated poker faces have an FID score of 100.38, the authors note that FID might not be a perfect metric for de-expression quality, as a failure to de-express can sometimes lead to a better FID score.
  • Future Work:
    • Investigate transferring expressions to tackle class imbalance issues.
  • Explore extending the method for plausible emotionless face synthesis.
  • Further research into the effectiveness of the emotional component for various downstream tasks.

Summary updated

Abstract

Representation learning and feature disentanglement have garnered significant research interest in the field of facial expression recognition (FER). The inherent ambiguity of emotion labels poses challenges for conventional supervised representation learning methods. Moreover, directly learning the mapping from a facial expression image to an emotion label lacks explicit supervision signals for capturing fine-grained facial features. In this article, we propose a novel FER model, named poker face vision transformer or PF-ViT, to address these challenges. PF-ViT aims to separate and recognize the disturbance-agnostic emotion from a static facial image by generating its corresponding poker face without the need for paired images. Inspired by the facial action coding system, we regard an expressive face as the combined result of a set of facial muscle movements on one's poker face (i.e., an emotionless face). PF-ViT utilizes vanilla vision transformers, and its components are first pretrained as masked autoencoders on a large facial expression dataset without emotion labels, yielding excellent representations. Subsequently, we train PF-ViT using a GAN framework. During training, the auxiliary task of poke face generation promotes the disentanglement between emotional and emotion-irrelevant components, guiding the FER model to holistically capture discriminative facial details. Quantitative and qualitative results demonstrate the effectiveness of our method, surpassing the state-of-the-art methods on four popular FER datasets.
BibTeX
@article{li2025emotion,
  author = {Jia Li and Jiantao Nie and Dan Guo and Richang Hong and Meng Wang},
  title = {{Emotion Separation and Recognition From a Facial Expression by Generating the Poker Face With Vision Transformers}},
  journal = {IEEE Trans. Comput. Soc. Syst.},
  year = {2025},
  doi = {10.1109/tcss.2024.3478839},
}
Figure