TL;DR
This research investigates how gender imbalances in training data lead to "hallucinated" biases in facial synthesis models (GANs). The study demonstrates that biased datasets cause face frontalization models to either fail to generate recognizable features or incorrectly change the subject's gender, identifying specific architectural features like skip connections that exacerbate these prejudices.
Introduction: The Problem of Generative Bias
While bias in image classification is well-documented, generative models have received far less scrutiny. This paper focuses on face frontalization: the process of synthesizing a front-facing view from a side-profile image. The core issue is that models trained on imbalanced datasets (e.g., male-dominated) don't just learn to rotate a face; they learn a "prototype" based on the dominant group. When processing a minority group member, the model may "correct" features to match the prototype or fail to reconstruct the face entirely, posing significant ethical risks for sensitive applications like security.
Methods and Frameworks
The researchers conducted a multi-stage experimental study using two Generative Adversarial Network (GAN) frameworks: Pix2Pix, a standard image-to-image baseline, and Pairwise-GAN, a state-of-the-art model utilizing dual U-Nets for left and right poses.
Key Analytical Techniques: * Latent Space Exploration: Analyzing variance in the semantic space to track how diversity is lost across layers. * Ablation Study: Modifying the Pairwise-GAN architecture by removing skip connections to test their role in transmitting bias. * Filter Analysis: Using Principal Component Analysis (PCA) on convolutional filters across 18 pre-trained models to visualize how "outlier" filters correlate with biased data.
Datasets and Pre-processing
The primary dataset was Color FERET, containing 11,338 images of 994 individuals. Images were cropped and aligned using MTCNN and Face++, specifically filtering for 67.5-degree side poses. The team constructed six training sets with male-to-female ratios ranging from 100:0 to 50:50 to observe incremental changes in model behavior against a balanced test set of 47 males and 47 females.
Key Insights
- The "50:50" Fallacy: Surprisingly, a perfectly balanced training set was not the most effective. The "top unbiased rate" of 84% was achieved at a different ratio, while the 50:50 ratio yielded only 71%, suggesting that fairness is about feature interaction, not just raw numbers.
- Bias Transmission: Skip connections, intended to preserve detail, were found to act as conduits for bias, passing "prejudiced" noise from the decoder back into the encoding process.
- Layer-Specific Vulnerability: Encoder layers 5 and 6 were identified as particularly sensitive to imbalanced data, showing significantly higher variance in biased models.
Key Results
- Gender Match Rate: In models trained only on males, there was a 91% probability of turning a female side-pose into a male frontal face. Even in balanced models, the gender match rate for females remained 20% lower than for males.
- Face Recovery Rate: When female representation in training dropped below 20%, the model's ability to generate any recognizable face plummeted.
- Architectural Mitigation: Removing skip connections in the Pairwise-GAN largely eliminated "failure in generation" (distorted faces), though correcting incorrect gender bias remains a more difficult challenge.
Limitations and Future Work
The authors acknowledge that evaluation tools like the Face++ API may have their own inherent biases. Future research will extend beyond gender to explore skin-color and age. Ultimately, the study suggests that while architectural changes can mitigate bias, complete elimination may be impossible without massive, perfectly curated datasets.
Summary updated