Demodalizing Face Recognition with Synthetic Samples

Zhonghua Zhai, Pengju Yang, Xiaofeng Zhang, Maji Huang, Haijing Cheng, Xuejun Yan, Chunmao Wang, Shiliang Pu

Abstract

Deep face recognition models require vast amounts of diverse data, often leading researchers to supplement real-world datasets with synthetic samples. However, training on synthetic data frequently degrades performance due to "unified modality information"—a mathematical signature unique to generators that causes models to overfit to style rather than identity. This paper proposes Demodalizing Face Recognition (DMFR), a framework designed to strip away these synthetic artifacts. By employing meta-learning, feature disentanglement, and explicit filtering, DMFR enables synthetic data to significantly enhance model robustness on challenging real-world benchmarks like IJB-C.

Introduction: The Synthetic Data Paradox

While synthetic face generation (via GANs or 3D modeling) offers a solution to the long-tailed distribution and biases of real-world datasets, its integration into training pipelines is notoriously difficult. Even when images appear realistic to the human eye, they contain consistent mathematical signatures—modalities—that deep neural networks easily exploit. Instead of learning robust identity features, models learn to recognize the generator's specific style. This leads to high inter-class similarity among synthetic images, where different individuals are grouped together based on their shared "synthetic look," ultimately harming generalization on real faces.

The DMFR framework aligns the feature distributions of real and synthetic samples, ensuring the model focuses on identity rather than the synthetic modality.

Methodology

The authors introduce three strategies for "demodalizing" synthetic data, ranging from implicit learning to explicit feature transformation.

1. Meta-learning-based DMFR

This approach treats real and synthetic data as distinct domains. Utilizing a meta-learning framework inspired by MAML (Model-Agnostic Meta-Learning), the model simulates domain shifts during training. By optimizing for performance on "meta-test" sets containing unseen domains, the model is forced to prioritize features that remain consistent across both real and synthetic modalities.

2. Disentangling-based DMFR

This method employs an encoder-decoder architecture to decompose face features into two distinct components: Identity ($f_{id}$) and Non-Identity ($f_{non-id}$). The non-identity component is supervised to capture synthetic artifacts, pose, and lighting, while the identity component is used for recognition. By isolating and discarding the synthetic modality, the model preserves only the information necessary for identification.

The disentangling-based DMFR architecture splits features into identity and non-identity components to isolate synthetic noise.

3. Filter-based DMFR

The most direct approach involves a "Filter" module that explicitly maps synthetic features into the real-image feature space. By minimizing the distance between a synthetic sample and its real-world counterpart, the filter "cleans" the synthetic signature before the features reach the classifier.

Experimental Evaluation

The framework was trained on a cleaned version of MS-Celeb-1M and evaluated on standard (LFW, MegaFace) and challenging (IJB-A, IJB-C) benchmarks. To specifically test robustness to pose and quality, the authors introduced two new benchmarks: CP-IJB-C (Cross-Pose) and CQ-IJB-C (Cross-Quality).

Key Results

DMFR demonstrated substantial gains in high-security scenarios (low False Acceptance Rates):

  • IJB-C Performance: On the IJB-C dataset at a FAR of $1e-4$, the baseline achieved a score of 0.766. While raw synthetic data often caused performance to drop, DMFR (disentangle) improved the score to 0.908.
  • Cross-Pose Robustness: On the CP-IJB-C benchmark, DMFR (disentangle) reached 0.659 at FAR $1e-4$, a significant leap from the baseline's 0.318. This indicates that the model successfully leveraged synthetic 3D poses without being distracted by their artificial nature.

Discussion and Insights

A critical insight of this work is the "Invisible Signature." Synthetic modalities are often imperceptible at the pixel level but prominent in high-dimensional feature space. Analysis shows that even minor synthetic modifications can cause massive shifts in feature magnitude. DMFR addresses this by reducing inter-class similarity—preventing the model from grouping different people together just because they are "synthetic"—while maintaining the intra-class diversity provided by the generator (e.g., varied poses).

Conclusion

Demodalizing Face Recognition provides a robust path for utilizing synthetic data in deep learning. By treating the synthetic signature as a removable modality, the DMFR framework allows models to benefit from the diversity of generated samples without falling victim to their artifacts. Future work may explore non-linear demodalization techniques to address more complex GAN-based distortions.

Summary updated

Abstract

Using data generated by generative adversarial networks or three-dimensional (3D) technology for face recognition training is a theoretically reasonable solution to the problems of unbalanced data distributions and data scarcity. However, due to the modal difference between synthetic data and real data, the direct use of data for training often leads to a decrease in the recognition performance, and the effect of synthetic data on recognition remains ambiguous. In this paper, after observing in experiments that modality information has a fixed form, we propose a demodalizing face recognition training architecture for the first time and provide a feasible method for recognition training using synthetic samples. Specifically, three different demodalizing training methods, from implicit to explicit, are proposed. These methods gradually reveal a generated modality that is difficult to quantify or describe. By removing the modalities of the synthetic data, the performance degradation is greatly alleviated. We validate the effectiveness of our approach on various benchmarks of large-scale face recognition and outperform the previous methods, especially in the low FAR range.
BibTeX
@inproceedings{zhai2021demodalizing,
  author = {Zhonghua Zhai and Pengju Yang and Xiaofeng Zhang and Maji Huang and Haijing Cheng and Xuejun Yan and Chunmao Wang and Shiliang Pu},
  title = {{Demodalizing Face Recognition with Synthetic Samples}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16439},
}
Figure