Abstract
Deep face recognition models require vast amounts of diverse data, often leading researchers to supplement real-world datasets with synthetic samples. However, training on synthetic data frequently degrades performance due to "unified modality information"—a mathematical signature unique to generators that causes models to overfit to style rather than identity. This paper proposes Demodalizing Face Recognition (DMFR), a framework designed to strip away these synthetic artifacts. By employing meta-learning, feature disentanglement, and explicit filtering, DMFR enables synthetic data to significantly enhance model robustness on challenging real-world benchmarks like IJB-C.
Introduction: The Synthetic Data Paradox
While synthetic face generation (via GANs or 3D modeling) offers a solution to the long-tailed distribution and biases of real-world datasets, its integration into training pipelines is notoriously difficult. Even when images appear realistic to the human eye, they contain consistent mathematical signatures—modalities—that deep neural networks easily exploit. Instead of learning robust identity features, models learn to recognize the generator's specific style. This leads to high inter-class similarity among synthetic images, where different individuals are grouped together based on their shared "synthetic look," ultimately harming generalization on real faces.
Methodology
The authors introduce three strategies for "demodalizing" synthetic data, ranging from implicit learning to explicit feature transformation.
1. Meta-learning-based DMFR
This approach treats real and synthetic data as distinct domains. Utilizing a meta-learning framework inspired by MAML (Model-Agnostic Meta-Learning), the model simulates domain shifts during training. By optimizing for performance on "meta-test" sets containing unseen domains, the model is forced to prioritize features that remain consistent across both real and synthetic modalities.
2. Disentangling-based DMFR
This method employs an encoder-decoder architecture to decompose face features into two distinct components: Identity ($f_{id}$) and Non-Identity ($f_{non-id}$). The non-identity component is supervised to capture synthetic artifacts, pose, and lighting, while the identity component is used for recognition. By isolating and discarding the synthetic modality, the model preserves only the information necessary for identification.
3. Filter-based DMFR
The most direct approach involves a "Filter" module that explicitly maps synthetic features into the real-image feature space. By minimizing the distance between a synthetic sample and its real-world counterpart, the filter "cleans" the synthetic signature before the features reach the classifier.
Experimental Evaluation
The framework was trained on a cleaned version of MS-Celeb-1M and evaluated on standard (LFW, MegaFace) and challenging (IJB-A, IJB-C) benchmarks. To specifically test robustness to pose and quality, the authors introduced two new benchmarks: CP-IJB-C (Cross-Pose) and CQ-IJB-C (Cross-Quality).
Key Results
DMFR demonstrated substantial gains in high-security scenarios (low False Acceptance Rates):
- IJB-C Performance: On the IJB-C dataset at a FAR of $1e-4$, the baseline achieved a score of 0.766. While raw synthetic data often caused performance to drop, DMFR (disentangle) improved the score to 0.908.
- Cross-Pose Robustness: On the CP-IJB-C benchmark, DMFR (disentangle) reached 0.659 at FAR $1e-4$, a significant leap from the baseline's 0.318. This indicates that the model successfully leveraged synthetic 3D poses without being distracted by their artificial nature.
Discussion and Insights
A critical insight of this work is the "Invisible Signature." Synthetic modalities are often imperceptible at the pixel level but prominent in high-dimensional feature space. Analysis shows that even minor synthetic modifications can cause massive shifts in feature magnitude. DMFR addresses this by reducing inter-class similarity—preventing the model from grouping different people together just because they are "synthetic"—while maintaining the intra-class diversity provided by the generator (e.g., varied poses).
Conclusion
Demodalizing Face Recognition provides a robust path for utilizing synthetic data in deep learning. By treating the synthetic signature as a removable modality, the DMFR framework allows models to benefit from the diversity of generated samples without falling victim to their artifacts. Future work may explore non-linear demodalization techniques to address more complex GAN-based distortions.
Summary updated