On Which Data Distribution (Synthetic or Real) We Should Rely for Soft Biometric Classification

Manju R. A, Atul Kumar, Akshay AgarwalWACV 2025

TL;DR

This paper investigates whether synthetic face images can effectively train deep learning models for gender classification, comparing their performance and robustness against models trained on real-world data. The study finds that while real-world data generally leads to better performance, Vision Transformers (ViTs) show remarkable resilience and maintain strong accuracy even when trained on synthetic data, offering a promising direction for privacy-preserving AI.

What Problem Does This Paper Address?

Gender classification is a crucial task in human-computer interaction and various other applications. Traditionally, this has relied on real-world facial images. However, collecting and labeling such datasets raises significant privacy concerns and can be challenging due to the sheer diversity of demographics and real-world conditions (e.g., varying lighting, pose, and occlusions). Generative AI has enabled the creation of synthetic face images, offering a potential solution to these challenges by providing large, privacy-friendly datasets. The core problem this paper addresses is understanding the effectiveness of these synthetic datasets for training gender classification models, specifically comparing them to real-world data and evaluating their robustness against various image corruptions.

Methods

The researchers implemented and evaluated several state-of-the-art deep learning architectures, focusing on two main categories: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Specifically, they used:

  • ViT Models: BEiT, ViT, Swin-T, and CViT. These models leverage self-attention mechanisms to process images.
  • CNN Models: ResNet-50, DenseNet-121, InceptionV3, and EfficientNet-B0. These models utilize convolutional layers for feature extraction.

To assess robustness, the models were tested on real-world datasets that were subjected to six types of common image corruptions: Color Saturation (CS), Color Contrast (CC), Blockwise Distortion (BW), Gaussian Noise in Color Component (GNC), Gaussian Blur (GB), and JPEG Compression (JPEG). The study also employed explainability techniques like Grad-CAM and t-SNE to visualize model attention and feature representations.

Datasets

The study utilized a combination of real and synthetic datasets:

  • Synthetic Dataset:
    • HDASynChildFace [17]: A synthetic dataset generated using StyleGAN3, designed to simulate various facial features across a wide age range. This dataset was used for training.
  • Real Datasets:
    • UTKFace [43]: A large-scale dataset with images covering a wide age range (0-116 years), annotated with age, gender, and ethnicity. A subset was used for training and testing.
  • AgeDB [27]: A dataset of facial images from celebrities, politicians, and scientists, used for testing.
  • FairFace [20]: A balanced dataset designed to address imbalances in other datasets, used for training and testing.

The real datasets were split into training and testing sets, with the combined training sets of UTKFace and FairFace used for training when real data was the focus. All three real datasets were used as test sets, both in their clean form and after applying perturbations.

Key Insights and Ideas

  • Real Data Reigns Supreme (Mostly): Models trained on real-world data consistently outperformed those trained on synthetic data across both ViT and CNN architectures. This suggests that real-world data still holds an edge in capturing the nuances of physical world distributions.
  • ViTs: The Privacy-Preserving Powerhouses: Vision Transformers, particularly BEiT, demonstrated remarkable robustness. They maintained high accuracy even when trained on synthetic data and were less sensitive to performance drops under various corruptions compared to CNNs. This makes them a strong candidate for privacy-conscious gender classification.
  • CNNs Struggle with Distribution Shift: CNN models exhibited a significant drop in performance when transitioning from real to synthetic training data, indicating a higher sensitivity to domain shift. Some CNNs, like DenseNet-121, performed close to random levels when trained on synthetic data.
  • Augmentation Bridges the Gap: Applying data augmentation techniques (using the same six perturbations) to the synthetic training data significantly improved model performance, bringing it closer to the performance achieved with real-world training data, especially for ViT models.
  • ViTs Understand "Where to Look": Grad-CAM visualizations showed that ViTs, even under distortion, consistently focused on critical facial regions (eyes, mouth), demonstrating effective attention mechanisms. This resilience was less pronounced in CNNs trained on synthetic data.

Key Results

  • Accuracy Advantage for Real Data: For the BEiT model, real data training yielded an average accuracy of 93.32%, while synthetic data training resulted in 87.75%. Similar trends were observed across all tested models.
  • ViT Robustness: The best performing ViT model, BEiT, showed a maximum accuracy drop of only 5.57% when trained on synthetic data compared to real data. In contrast, CNN models could see drops of up to 28.14%.
  • Perturbation Sensitivity: All models, both ViT and CNN, showed vulnerability to JPEG compression. Gaussian Noise in Color Component (GNC) also impacted performance significantly.
  • Augmentation Impact: For BEiT, augmentation on synthetic data improved accuracy from 88.53% to 90.69% on UTKFace, narrowing the gap with real-data training (94.45%).

Limitations and Future Work

  • Privacy vs. Performance Trade-off: While synthetic data offers privacy benefits, real-world data still provides superior performance. The study highlights ViTs as a promising compromise.
  • Model Sensitivity to Corruption: Despite improvements, models still showed sensitivity to certain corruptions, particularly JPEG compression.
  • Future Work: The authors suggest further exploring data augmentation strategies and investigating other generative models for synthetic data creation. They also point to the need for more research into understanding the specific reasons behind the performance disparity between real and synthetic data and how to fully bridge this gap.

Summary updated

BibTeX
@inproceedings{a2025which,
  author = {Manju R. A and Atul Kumar and Akshay Agarwal},
  title = {{On Which Data Distribution (Synthetic or Real) We Should Rely for Soft Biometric Classification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure