More comprehensive facial inversion for more effective expression recognition

Jiawei Mao, Guangyi Zhao, Yuanqi Chang, Xuesong Yin, Xiaogang Peng, Rui XuarXiv 2022 · DOI · arXiv

Abstract

Facial Expression Recognition (FER) remains a challenging task in human-computer interaction due to the subtle and holistic nature of emotional manifestations. We present Inversion FER (IFER), a novel framework that enhances expression recognition by integrating facial image inversion. At its core is the Adversarial Style Inversion Transformer (ASIT), which utilizes long-range dependencies to capture both global facial structures and local details. Our approach demonstrates that the process of reconstructing a face provides a rich feature set for classification, leading to state-of-the-art performance across multiple benchmarks.

Introduction

Traditional FER models primarily function as discriminative classifiers, often focusing on isolated facial patches like the eyes or mouth. However, emotions are expressed through complex, global changes in facial geometry. Current methods struggle to bridge the gap between local feature extraction and global context. We propose that the underlying structural information used in image generation can significantly bolster recognition accuracy. By learning to "invert" an image—mapping it back to a generative latent space—a model gains a deeper understanding of facial semantics.

Methodology

The IFER framework consists of two main pillars: the ASIT architecture and the inversion-based recognition task.

Adversarial Style Inversion Transformer (ASIT)

Unlike standard CNN-based inversion models, ASIT employs a pure Transformer architecture. This design allows for the modeling of long-range dependencies, ensuring the system considers the entire face simultaneously rather than just local neighborhoods. The model uses a window-based attention mechanism that transitions from a broad global view to fine-grained local details.

Heatmaps showing facial regions prioritized for expression recognition across different emotions.

Learning through Inversion

To ensure semantic fidelity, we utilize an Image Inversion Discriminator that measures cosine similarity between source and generated features. Furthermore, a Distribution Alignment Loss is applied, treating a pre-trained StyleGAN2 generator as a teacher to guide the ASIT encoder. This process forces the encoder to align its internal feature distributions with a high-quality latent space.

Key Insights

  • Generative Features for Recognition: The mapping of an image back into a generative latent space acts as a powerful feature extractor. These "latent codes" and "structural codes" capture the intrinsic geometry of an expression more effectively than standard classification layers.
  • Global Context via Transformers: By moving away from CNNs, the model avoids over-relying on single features (e.g., just the mouth for a smile) and instead analyzes the coordinated movement of the entire face.
  • Structural Integrity: Ablation studies reveal that "Structure Codes"—representing high-level semantics—are essential for maintaining the geometric "skeleton" of an expression.

Results and Evaluation

The model was evaluated on high-standard datasets including FFHQ and CelebA-HQ for inversion, and RAF-DB, SFEW, and AffectNet for recognition.

Qualitative results of facial image inversion, comparing original images (left) with ASIT reconstructions (right).

ASIT achieved state-of-the-art inversion quality, outperforming models like pSp and e4e in terms of MSE and FID. In the FER domain, IFER reached an accuracy of 87.23% on the RAF-DB dataset, surpassing specialized models such as TransFER. Notably, removing the inversion pre-training stage led to a 4% drop in accuracy, confirming that the generative task is a critical driver of recognition performance.

Conclusion and Future Work

IFER demonstrates that "learning to generate" is a viable path toward "learning to recognize." While the computational cost of Transformers and StyleGAN2 remains a hurdle for mobile deployment, the gain in accuracy and structural understanding is significant. Future research will look toward extending this framework to video-based expression inversion and improving robustness against extreme head poses and occlusions.

Summary updated

Abstract

Facial expression recognition (FER) plays a significant role in the ubiquitous application of computer vision. We revisit this problem with a new perspective on whether it can acquire useful representations that improve FER performance in the image generation process, and propose a novel generative method based on the image inversion mechanism for the FER task, termed Inversion FER (IFER). Particularly, we devise a novel Adversarial Style Inversion Transformer (ASIT) towards IFER to comprehensively extract features of generated facial images. In addition, ASIT is equipped with an image inversion discriminator that measures the cosine similarity of semantic features between source and generated images, constrained by a distribution alignment loss. Finally, we introduce a feature modulation module to fuse the structural code and latent codes from ASIT for the subsequent FER work. We extensively evaluate ASIT on facial datasets such as FFHQ and CelebA-HQ, showing that our approach achieves state-of-the-art facial inversion performance. IFER also achieves competitive results in facial expression recognition datasets such as RAF-DB, SFEW and AffectNet. The code and models are available at https://github.com/Talented-Q/IFER-master.
BibTeX
@misc{mao2022more,
  author = {Jiawei Mao and Guangyi Zhao and Yuanqi Chang and Xuesong Yin and Xiaogang Peng and Rui Xu},
  title = {{More comprehensive facial inversion for more effective expression recognition}},
  year = {2022},
  eprint = {2211.13564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.13564},
}
Figure