Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization

Yuhang Li, Xin Dong, Chen Chen, Jingtao Li, Yuxin Wen, Michael Spranger, Lingjuan Lyu

Abstract

Transfer learning from large-scale datasets like ImageNet is standard practice, but acquiring sufficient real-world data for niche downstream tasks remains a challenge. While synthetic data from generative models like Stable Diffusion offers a potential solution, naive "Mixed Transfer" often degrades performance due to the distribution gap. This paper introduces Bridged Transfer, a two-stage pipeline that uses synthetic data to warm up feature extractors, and Dataset Style Inversion (DSI) to align synthetic images with target domains. Our results demonstrate significant accuracy improvements, particularly in data-scarce few-shot scenarios.

Introduction

Transfer learning—taking a model pre-trained on a massive dataset like ImageNet and fine-tuning it for a specific task—is the industry standard. However, obtaining high-quality real-world data for niche tasks (e.g., identifying specific aircraft models) is often expensive or restricted by privacy and copyright.

Generative AI, specifically models like Stable Diffusion, offers a potential solution: prompting a model to create the required training data. However, a significant distribution gap exists. Even high-fidelity synthetic images differ mathematically from real photos. The authors found that "Mixed Transfer"—simply combining synthetic and real images—leads to a 6–10% drop in accuracy as synthetic artifacts "pollute" the model’s feature representation.

Bridged Transfer and Dataset Style Inversion

To overcome the distribution gap, the authors propose a two-pronged approach:

  1. Bridged Transfer (and Bridged Transfer++): This is a two-stage adaptation process. In Stage 1, the ImageNet pre-trained model is fine-tuned using only synthetic images. This allows the feature extractor to learn the shapes and textures of the target domain without being misled by the specific labels of the synthetic data. In Stage 2, the model is adapted using only real images. The "++" version involves re-initializing the fully connected (FC) layer before Stage 2 and using Mixup regularization to prevent the classifier from over-associating synthetic artifacts with class labels.
  2. Dataset Style Inversion (DSI): Generic prompts often fail to capture the specific visual "vibe" (lighting, camera angle, texture) of a target dataset. DSI optimizes a single "style token" ($S^*$) that represents the entire target dataset's style. Prompts like "A photo of a [Class] in the style of $S^*$" generate images that are more compatible with the real-world target data.
Overview of the Bridged Transfer pipeline using synthetic data to bridge the gap between pre-training and downstream tasks.

Experimental Evaluation

The researchers evaluated their methods across 10 diverse downstream datasets, including fine-grained objects (Aircraft, Cars, Birds, Dogs, Pets, Flowers), general scenes (Caltech-101, Food-101, SUN397), and textures (DTD). They tested architectures ranging from ResNet-18/50 to Vision Transformers (ViT-B/16, ViT-L/16).

Key Results and Insights

The proposed methods yielded substantial improvements across the board:

  • Few-Shot Dominance: The gains are most dramatic when real data is scarce. In a 4-shot scenario (only 4 real images per class), Bridged Transfer++ improved accuracy on the Stanford Cars dataset by 60% compared to standard transfer learning.
  • Classifier as the Weak Link: A critical insight is that while synthetic data helps the feature extractor ("the eyes"), it can confuse the classifier ("the brain"). Re-initializing the classifier before the second stage of training is essential to prevent performance degradation.
  • Scalability: Unlike many data augmentation techniques that plateau, the benefits of synthetic data showed no signs of saturation even at 3,000 images per class.
  • Style over Content: DSI consistently added a 0.5% to 3% accuracy boost over simple text templates, proving that matching the domain's style is as important as the object content itself.
Performance comparison across various datasets showing the superiority of Bridged Transfer++ in low-data (few-shot) regimes.

Conclusion and Future Work

Bridged Transfer provides a robust framework for utilizing synthetic data in transfer learning. By treating synthetic data as a "bridge" rather than a direct supplement, models can achieve faster convergence and higher final accuracy. While the computational cost of generating and training on thousands of images is a factor, the performance gains—especially in few-shot settings—are significant. Future work may explore extending these "bridges" to more complex tasks like object detection and segmentation.

Summary updated

Abstract

Synthetic image data generation represents a promising avenue for training deep learning models, particularly in the realm of transfer learning, where obtaining real images within a specific domain can be prohibitively expensive due to privacy and intellectual property considerations. This work delves into the generation and utilization of synthetic images derived from text-to-image generative models in facilitating transfer learning paradigms. Despite the high visual fidelity of the generated images, we observe that their naive incorporation into existing real-image datasets does not consistently enhance model performance due to the inherent distribution gap between synthetic and real images. To address this issue, we introduce a novel two-stage framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability and subsequently uses real data for rapid adaptation. Alongside, We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements, with up to 30% accuracy increase on classification tasks. Intriguingly, we note that the enhancements were not yet saturated, indicating that the benefits may further increase with an expanded volume of synthetic data.
BibTeX
@misc{li2024is,
  author = {Yuhang Li and Xin Dong and Chen Chen and Jingtao Li and Yuxin Wen and Michael Spranger and Lingjuan Lyu},
  title = {{Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization}},
  year = {2024},
  eprint = {2403.19866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.19866},
}
Figure