TL;DR
This paper investigates why synthetic images generated by deep generative models (DGMs) often lead to performance drops in downstream tasks like image classification, even when they look photorealistic. The authors propose a new taxonomy for the "Domain Gap" and find that the "Content Gap" is the primary culprit. They then introduce two novel regularization strategies, Pretrained Guidance and Real Guidance, to effectively leverage synthetic data, especially in low-data scenarios.
What Problem Does This Paper Address?
Deep generative models (DGMs) like GANs and diffusion models have become incredibly adept at producing high-fidelity, photorealistic images. This has led to excitement about using these synthetic images to augment real datasets, particularly in domains where acquiring real data is difficult or expensive. However, a persistent problem is that training models on purely synthetic data often results in a significant performance degradation compared to training on real data. This "domain gap" is a known issue, but the paper argues that existing explanations are too general. The authors aim to dissect the domain gap into its constituent factors and understand which ones are most detrimental to downstream task performance, ultimately proposing methods to mitigate these issues and make synthetic data more useful.
Methods
The paper introduces a new taxonomy to categorize the factors contributing to the domain gap:
- Appearance Gap: Differences in visual textures or lighting.
- Content Gap: Distributional shifts in the composition or attributes of objects within images.
- Quality Gap: Structural distortions or artifacts in synthetic images.
To address the performance drop, they propose two novel regularization strategies:
- Pretrained Guidance (PG): This method uses a frozen, pretrained model (e.g., trained on ImageNet) to guide the training of a new model on synthetic data. It encourages the new model's feature representations to maintain a similar span or distribution of distances as those from the pretrained model, preventing it from collapsing into overly simplistic representations. This is achieved by regularizing the distance matrices of features from both models.
- Real Guidance (RG): Inspired by continual learning, this strategy uses a small set of real data to regularize the gradients generated by synthetic data. It ensures that synthetic data gradients are at most perpendicular to real data gradients, preventing misleading signals from synthetic data from negatively impacting the model's learning.
Datasets
The paper utilizes several datasets for their experiments:
- CIFAR-10: Used extensively for initial investigations and as a benchmark for synthetic-to-real generalization and data augmentation.
- CUB Bird: A dataset for fine-grained image classification.
- Oxford Flower: Another dataset for fine-grained image classification.
- SDI: An industrial dataset for defect detection.
Synthetic images were generated using popular DGMs such as LSGM, SG-XL, Projected GAN (Proj-GAN), and DT-GAN.
Key Insights and Ideas
- The "Content Gap" is the Main Culprit: Contrary to what might be assumed, the paper's experiments suggest that the primary reason for performance degradation isn't just unrealistic textures (Appearance Gap) or image distortions (Quality Gap), but rather the DGM's failure to capture the full diversity of the real data distribution, especially rare attributes or compositions (Content Gap).
- Non-Mutual Domain Gap: The performance drop is not symmetrical. Models trained on real data tend to perform reasonably well on synthetic data, but the reverse is not true. This highlights that synthetic data is often a simplified subset of real data.
- Synthetic Data Training Saturates Faster: Classifiers trained on synthetic data reach their peak accuracy much faster but also plateau at a lower level, suggesting they are learning a less complex or diverse set of features.
- Low-Loss Synthetic Samples Still Help: Even synthetic samples that produce very low loss when evaluated on a real-data-trained classifier (indicating they are "easy" for the model to classify) can still provide some benefit when used for data augmentation.
- Pretrained Guidance is Flexible: The proposed Pretrained Guidance method doesn't require the pretrained and target models to have identical architectures, making it adaptable to various scenarios, including using a standard ImageNet-pretrained model to guide a custom architecture for a specific task.
Key Results
- Improved Zero-Shot Performance: The proposed Pretrained Guidance (PG) method significantly outperforms existing baselines in zero-shot classification tasks (where no real data is available for downstream training), achieving higher accuracies across various datasets. For instance, on CIFAR-10, their PG-R method achieved 85.59% accuracy, outperforming baselines like ASG [8] (82.47%) and CSG [7] (82.67%) when trained on LSGM synthetic data.
- Effective Data Augmentation: When used as data augmentation in low-shot scenarios, the combination of Pretrained Guidance and Real Guidance (PG+RG) consistently improved performance over baselines. For example, on CIFAR-10 (10-shots), their method achieved 86.58% accuracy, significantly better than the baseline data augmentation (85.57%) and A-GEM (86.91%).
- Robustness to Initialization: The Pretrained Guidance method showed strong performance even when the target model (Mu) was not initialized with ImageNet weights, demonstrating its effectiveness beyond simple initialization benefits.
Limitations and Future Work
- Mode Dropping on Small Datasets: The paper acknowledges that DGMs trained on small datasets can suffer from mode dropping, where rare samples are omitted. This is a core motivation for their work, but it implies that the effectiveness of their methods might be further limited if the DGM itself fails to learn certain crucial modes.
- Hyperparameter Tuning: The effectiveness of the proposed regularizations (PG and RG) depends on hyperparameters ($\lambda_1, \lambda_2, \lambda_3$), which need to be tuned for different datasets and tasks.
- Future Work: The authors suggest exploring finetuning DGMs on smaller datasets as a potential avenue, which requires careful adaptation. They also mention that their domain adaptation methods primarily tackle significant distribution shifts, and further research is needed for fine-grained shifts between DGM synthetic data and real images.
Summary updated