Abstract
Training robust facial analysis systems requires massive datasets with diverse lighting and expressions, yet collecting such data in-the-wild often results in long-tail distributions. Generative Adversarial Networks (GANs) offer a solution through synthetic data augmentation, but they frequently suffer from attribute entanglement and poor perceptual quality. This paper introduces LEGAN (Lighting-Expression GAN), a novel framework that jointly manipulates directional lighting and facial expressions while strictly preserving subject identity. By integrating a "Quality Estimation Model" trained on human perceptual ratings, LEGAN ensures synthetic images appear natural to the human eye. Experimental results show that LEGAN outperforms state-of-the-art models in image quality and identity preservation, while significantly boosting the performance of downstream facial recognition and emotion detection tasks.
Introduction
Facial analysis systems, such as emotion detection and face recognition, are highly sensitive to variations in lighting and expression. Obtaining balanced datasets that cover extreme conditions (e.g., harsh side-lighting or intense expressions) is a significant challenge in computer vision. While synthetic data generation via GANs is a promising alternative, existing methods often fail in two key areas: entanglement and perceptual realism. Entanglement occurs when changing one attribute (like an expression) inadvertently alters another (like lighting or identity). Furthermore, standard automated metrics like the Fréchet Inception Distance (FID) often fail to capture the "uncanny valley" artifacts—such as "bullet-hole" textures—that make synthetic faces appear unnatural to human observers.
Methods
LEGAN addresses these issues through a multi-stage architecture and a human-centric loss function. The framework utilizes three parallel "hourglass" (encoder-decoder) networks. Two networks generate transformation masks—one for lighting and one for expression—which are then combined with the original image by a third network to synthesize the final output. This mask-based approach ensures that edits are localized and do not bleed into unrelated attributes.
The Quality Estimation Model (Q)
The core innovation of LEGAN is the Quality Estimation Model (Q). Recognizing that mathematical metrics do not always align with human perception, the authors trained a separate CNN to predict the "naturalness" of a synthetic face. This model was trained on approximately 37,000 synthetic images from various sources (e.g., StyleGAN, DeepFakes) which were rated by human workers on Amazon Mechanical Turk. During LEGAN's training, this pre-trained "Q" model acts as an auxiliary discriminator, forcing the generator to produce images that score high on the human naturalness scale.
Loss Functions
The model optimizes a composite loss function including:
- Adversarial Loss: To ensure general image realism.
- Classification Loss: To verify the target attribute (lighting or expression) is achieved.
- Identity Loss: Using a pre-trained face recognition network to ensure the subject remains the same.
- Perceptual Quality Loss: Derived from the Q-model to eliminate artifacts.
Key Results
LEGAN was evaluated against popular models like StarGAN and StarGAN-v2 using the CMU-MultiPIE dataset. Quantitatively, LEGAN achieved a superior FID of 29.96 (compared to StarGAN's 38.74) and a higher identity match score. In a perceptual study, human raters preferred LEGAN-generated images 39.3% of the time, a significant margin over its competitors.
Beyond image quality, the paper demonstrates the utility of LEGAN for data augmentation. Using LEGAN-generated images to augment training sets for face verification (IJB-B) improved the True Positive Rate from 0.954 to 0.967. Similarly, it boosted accuracy for under-represented classes in expression classification on the AffectNet dataset.
Conclusion
LEGAN successfully bridges the gap between mathematical plausibility and human perceptual quality in facial synthesis. By "distilling" human judgment into a Quality Estimation Model, the framework produces high-quality, disentangled manipulations that are effective for training real-world AI models. Future work will explore temporal consistency for video synthesis and more granular control over facial muscles using Action Units (AUs).
Summary updated