Abstract
Transfer learning from large-scale datasets like ImageNet is standard practice, but acquiring sufficient real-world data for niche downstream tasks remains a challenge. While synthetic data from generative models like Stable Diffusion offers a potential solution, naive "Mixed Transfer" often degrades performance due to the distribution gap. This paper introduces Bridged Transfer, a two-stage pipeline that uses synthetic data to warm up feature extractors, and Dataset Style Inversion (DSI) to align synthetic images with target domains. Our results demonstrate significant accuracy improvements, particularly in data-scarce few-shot scenarios.
Introduction
Transfer learning—taking a model pre-trained on a massive dataset like ImageNet and fine-tuning it for a specific task—is the industry standard. However, obtaining high-quality real-world data for niche tasks (e.g., identifying specific aircraft models) is often expensive or restricted by privacy and copyright.
Generative AI, specifically models like Stable Diffusion, offers a potential solution: prompting a model to create the required training data. However, a significant distribution gap exists. Even high-fidelity synthetic images differ mathematically from real photos. The authors found that "Mixed Transfer"—simply combining synthetic and real images—leads to a 6–10% drop in accuracy as synthetic artifacts "pollute" the model’s feature representation.
Bridged Transfer and Dataset Style Inversion
To overcome the distribution gap, the authors propose a two-pronged approach:
- Bridged Transfer (and Bridged Transfer++): This is a two-stage adaptation process. In Stage 1, the ImageNet pre-trained model is fine-tuned using only synthetic images. This allows the feature extractor to learn the shapes and textures of the target domain without being misled by the specific labels of the synthetic data. In Stage 2, the model is adapted using only real images. The "++" version involves re-initializing the fully connected (FC) layer before Stage 2 and using Mixup regularization to prevent the classifier from over-associating synthetic artifacts with class labels.
- Dataset Style Inversion (DSI): Generic prompts often fail to capture the specific visual "vibe" (lighting, camera angle, texture) of a target dataset. DSI optimizes a single "style token" ($S^*$) that represents the entire target dataset's style. Prompts like "A photo of a [Class] in the style of $S^*$" generate images that are more compatible with the real-world target data.
Experimental Evaluation
The researchers evaluated their methods across 10 diverse downstream datasets, including fine-grained objects (Aircraft, Cars, Birds, Dogs, Pets, Flowers), general scenes (Caltech-101, Food-101, SUN397), and textures (DTD). They tested architectures ranging from ResNet-18/50 to Vision Transformers (ViT-B/16, ViT-L/16).
Key Results and Insights
The proposed methods yielded substantial improvements across the board:
- Few-Shot Dominance: The gains are most dramatic when real data is scarce. In a 4-shot scenario (only 4 real images per class), Bridged Transfer++ improved accuracy on the Stanford Cars dataset by 60% compared to standard transfer learning.
- Classifier as the Weak Link: A critical insight is that while synthetic data helps the feature extractor ("the eyes"), it can confuse the classifier ("the brain"). Re-initializing the classifier before the second stage of training is essential to prevent performance degradation.
- Scalability: Unlike many data augmentation techniques that plateau, the benefits of synthetic data showed no signs of saturation even at 3,000 images per class.
- Style over Content: DSI consistently added a 0.5% to 3% accuracy boost over simple text templates, proving that matching the domain's style is as important as the object content itself.
Conclusion and Future Work
Bridged Transfer provides a robust framework for utilizing synthetic data in transfer learning. By treating synthetic data as a "bridge" rather than a direct supplement, models can achieve faster convergence and higher final accuracy. While the computational cost of generating and training on thousands of images is a factor, the performance gains—especially in few-shot settings—are significant. Future work may explore extending these "bridges" to more complex tasks like object detection and segmentation.
Summary updated