IS SYNTHETIC DATA USEFUL FOR TRANSFER LEARNING? AN INVESTIGATION INTO DATA GENERATION, VOLUME, AND UTILIZATION

Yuhang Li, Xin Dong, Chen Chen, Jingtao Li, Yuxin Wen, Michael Spranger, Lingjuan LyuSubmission to ICLR 2024 · Rejected

TL;DR

Simply mixing text-to-image synthetic data with real images hurts transfer learning performance due to distribution mismatch. To unlock the value of generative models, this paper introduces Bridged Transfer++—a two-stage pipeline that pre-tunes representations on synthetic data before adapting to real images—paired with Dataset Style Inversion (DSI) to stylistically align synthetic images with downstream domains.

What Problem Does This Paper Address?

Transfer learning typically involves pre-training a neural network on a massive dataset (like ImageNet) and fine-tuning it on a specialized downstream task. However, collecting real-world data in niche or fine-grained domains is often bottlenecked by privacy, copyright, or expense.

Modern text-to-image models like Stable Diffusion can generate millions of photorealistic images, offering an appealing source of data augmentation. Yet, naively throwing synthetic images into real downstream datasets ("Mixed Transfer") consistently degrades classification accuracy by 6–10%. The authors set out to determine: Can synthetic images genuinely benefit downstream transfer learning, and if so, how should we generate, scale, and utilize them?

Methods

The authors investigate and mitigate the distribution gap between synthetic and real images through three core contributions:

  1. Bridged Transfer: Rather than co-training on a mixture of real and synthetic images, the authors propose a sequential two-stage fine-tuning pipeline:
  2. Stage 1: Fine-tune the pre-trained ImageNet model purely on synthetic data.
  3. Stage 2: Rapidly adapt the fine-tuned model using only downstream real images.
  4. Bridged Transfer++ (Regularization & Classifier Reset): The authors discovered that while Stage 1 builds a significantly better feature representation, the final linear classification head overfits to synthetic generation artifacts. To solve this:
  5. Mixup loss is applied during Stage 1 to produce a more generalizable feature extractor.
  6. FC Reinitialization (FC Reinit): The final classifier head is discarded and randomly reinitialized before Stage 2 fine-tuning on real data.
  7. Dataset Style Inversion (DSI): To stylistically align synthetic images with the target domain without expensive per-class inversion, DSI optimizes a single pseudo-word token ($S^*$) representing the entire downstream dataset's style via diffusion latent optimization. Class prompts then follow: "A $C_i$ photo in the style of $S^*$."
ImageNet Pre-trained Model 
      │
      ▼ (Stage 1: Fine-tune on Synthetic Images + Mixup)
Feature Extractor + Overfitted Classifier
      │
      ▼ (Discard & Reinitialize FC Classifier)
Pre-tuned Feature Extractor + Fresh Classifier
      │
      ▼ (Stage 2: Fast Adaptation on Real Images)
Optimized Downstream Model

Datasets

  • Pre-training: ImageNet-1K.
  • Evaluation Benchmarks (10 downstream datasets): FGVC Aircraft, Caltech-101, Stanford Cars, CUB-200-2011, Describable Textures Dataset (DTD), Oxford Flowers-102, Food-101, Oxford-IIIT Pets, Stanford Dogs, and SUN397.
  • Synthetic Data: Generated using Stable Diffusion v1.5 (500 to 3,000 images per class).

Key Insights and Ideas

  • Naive data pooling fails: Directly blending real and synthetic data distorts the underlying distribution, leading to poorer decision boundaries on real test sets.
  • The feature extractor learns; the classifier gets fooled: Transferability metrics (LEEP score) and training convergence curves reveal that synthetic data makes representations much more transferable. However, the classifier layer rapidly memorizes synthetic quirks, associating generation artifacts with class labels. Reinitializing the classifier head completely resolves this bottleneck.
  • Data volume hasn't hit saturation: Scaling synthetic samples from 500 up to 3,000 images per class yields monotonic accuracy gains across tasks, indicating further headroom with even larger synthetic pools.
  • Single-token style conditioning is fast and effective: DSI captures domain nuances (e.g., ensuring DTD’s "braided" class generates braided textures rather than human hair braids) while requiring only one training run per dataset (20k steps), unlike class-wise textual inversion which scales linearly with class count.

Key Results

  • Full-Shot Transfer: Across all 10 datasets using ResNet-18, Bridged Transfer++ consistently outperforms standard fine-tuning, with notable gains on fine-grained benchmarks like FGVC Aircraft (+5.5% to 85.2%) and Stanford Cars (+7.8% to 91.6%).
  • Few-Shot Regimes: Improvements are massive in data-starved setups: on 4-shot Stanford Cars, Bridged Transfer++ delivers an absolute accuracy boost of up to ~60%.
  • Architectural Generalization: Testing on ResNet-50, ViT-B/16, and ViT-L/16 in a 4-shot regime yielded average accuracy jumps of 13%, 12%, and 9%, respectively (including a 53% improvement for ViT-B/16 on Stanford Cars).
  • Style Alignment: DSI consistently improved over plain template prompts (e.g., SUN397 accuracy improved from 60.8% to 63.4%).

Limitations and Future Work

  • Computational Cost: Generating thousands of images per class and running a two-stage full-network fine-tuning routine introduces notable computational and storage overhead.
  • Scope of Tasks: Experiments are confined to image classification; downstream transfer to dense prediction tasks (detection, segmentation) remains untested.
  • Scaling Limits: Because accuracy did not saturate at 3,000 synthetic images per class, the upper bounds and scaling laws of synthetic pre-transfer training remain an open question.

Summary updated

Abstract

Synthetic image data generation represents a promising avenue for training deep learning models, particularly in the realm of transfer learning, where obtaining real images within a specific domain can be prohibitively expensive due to privacy and intellectual property considerations. This work delves into the generation and utilization of synthetic images derived from text-to-image generative models in facilitating transfer learning paradigms. Despite the high visual fidelity of the generated images, we observe that their naive incorporation into existing real-image datasets does not consistently enhance model performance due to the inherent distribution gap between synthetic and real images. To address this issue, we introduce a novel two-stage framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability and subsequently uses real data for rapid adaptation. Alongside, We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements, with up to 30% accuracy increase on classification tasks. Intriguingly, we note that the enhancements were not yet saturated, indicating that the benefits may further increase with an expanded volume of synthetic
BibTeX
@misc{li2024is,
  author = {Yuhang Li and Xin Dong and Chen Chen and Jingtao Li and Yuxin Wen and Michael Spranger and Lingjuan Lyu},
  title = {{IS SYNTHETIC DATA USEFUL FOR TRANSFER LEARNING? AN INVESTIGATION INTO DATA GENERATION, VOLUME, AND UTILIZATION}},
  year = {2024},
  eprint = {2403.19866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.19866},
}
Figure