Abstract
Deep learning models for Facial Expression Recognition (FER) are often limited by the lack of large-scale, diversely labeled training data. This paper proposes a unified framework that jointly optimizes facial expression synthesis and recognition. By employing a Facial Expression Synthesis GAN (FESGAN) alongside a recognition network, the system creates a "Guide and Boost" loop where synthetic data enhances the recognizer, and recognition feedback improves the generator. To address the inherent distribution shift between real and synthetic data, a novel Real Data-guided Back-Propagation (RDBP) algorithm is introduced. The proposed method achieves state-of-the-art results on several prominent FER benchmarks, including CK+, Oulu-CASIA, and MMI.
Introduction
Facial Expression Recognition (FER) is a critical task in computer vision with applications ranging from human-computer interaction to psychological analysis. However, the performance of deep FER models is heavily dependent on the availability of large, high-quality datasets. Existing datasets are often small, expensive to annotate, and lack identity diversity.
While Generative Adversarial Networks (GANs) offer a potential solution by synthesizing additional training data, they introduce two significant challenges: Data Bias and Identity Preservation. Synthetic images often reside in a different feature space than real images, leading to poor generalization. Furthermore, maintaining consistent identity while varying expressions is technically demanding. This paper addresses these issues through a joint learning approach.
Methodology
The proposed FESR framework consists of two main components: a synthesis network (FESGAN) and a recognition network, optimized through a multi-stage process.
FESGAN: Identity-Preserving Synthesis
FESGAN is pre-trained to generate facial images with new identities by sampling from a prior distribution. Unlike traditional GANs that perform expression transfer on existing faces, FESGAN focuses on identity diversity. It incorporates an identity-preserving loss based on a pre-trained Light CNN-29 to ensure that synthesized faces maintain human-like consistency across different expressions.
Joint Optimization and the "Guide and Boost" Loop
In the second stage, the recognition network and the FESGAN generator are trained simultaneously. This creates a symbiotic relationship:
- Guide: The classification loss from the recognition network is back-propagated to the generator, guiding it to produce more discriminative and recognizable expressions.
- Boost: The generator provides a virtually infinite stream of diverse training data, boosting the recognition network's robustness.
Real Data-guided Back-Propagation (RDBP)
To bridge the gap between real and synthetic data distributions, the authors introduce an intra-class loss and the RDBP algorithm. RDBP treats real image features as fixed "anchors" in the feature space.
RDBP forces synthetic image features to align with the distribution of real images. Crucially, it prevents synthetic noise from "polluting" the recognition network's understanding of real faces by only updating the network's parameters relative to synthetic inputs while keeping the real data distribution as the ground truth.
Experimental Results
The FESR method was evaluated on five major facial expression databases: CK+, Oulu-CASIA, MMI, Multi-PIE, and TFD.
- CK+: Achieved a state-of-the-art accuracy of 99.34%, surpassing previous benchmarks.
- MMI: Reached 84.81% accuracy, a substantial improvement over previous methods like DeRL (73.23%), demonstrating effectiveness on challenging, non-uniform poses.
- Oulu-CASIA: Achieved 88.13% accuracy.
- Robustness: Tests on the Multi-PIE dataset showed superior performance across various camera angles (up to ±30°), confirming the model's resilience to viewpoint changes.
Conclusion
The joint learning of synthesis and recognition provides a powerful mechanism for overcoming data scarcity in FER. By using the recognition network to guide the GAN and the GAN to boost the recognizer, the FESR framework achieves superior performance. The RDBP algorithm effectively mitigates data bias, ensuring that synthetic data translates into real-world accuracy. Future work will focus on improving the visual quality of subtle expressions and extending the framework to video-based recognition.
Summary updated