TL;DR
The paper introduces SGID (Semantic-guided Generative Image Augmentation), a novel data augmentation method that leverages diffusion models to generate highly diverse training images while strictly preserving their original semantic meaning. By combining groundtruth labels and AI-generated captions as textual guidance, SGID strikes an optimal balance between image diversity and semantic consistency, leading to significant accuracy boosts across various vision backbones and datasets.
What Problem Does This Paper Address?
Data augmentation (DA) is essential for training deep neural networks, especially when labeled data is scarce. Traditional DA techniques generally fall into two camps, both with distinct bottlenecks:
- Perturbation-based methods (e.g., CutMix, Random Erasing) modify local regions of an image. While they preserve the original semantics perfectly, they lack visual diversity.
- Generative methods (e.g., Text2Img via Diffusion Models) synthesize entirely new images. They offer incredible diversity but frequently suffer from semantic drift—altering crucial details like adding an extra door handle to a car or changing its brand badge, which misleads the image classifier during training.
SGID addresses this trade-off by introducing a framework that injects explicit semantic guidance into an image-to-image diffusion pipeline, ensuring the generated images are both highly diverse and semantically accurate.
Methods
SGID splits the generative augmentation process into two main steps:
1. Label Acquisition and Caption Generation
To capture the exact semantics of an original image $x$, SGID combines its groundtruth class label with a descriptive caption:
- Label Prompt: The class label is converted into a simple sentence (e.g., "A picture of a [Chevrolet Silverado]").
- Caption Generation: The BLIP model generates a pool of candidate captions for the image using nucleus sampling or beam search.
- CLIP Filtering: A pre-trained CLIP model ranks these captions based on their cosine similarity to the original image, selecting the highest-scoring caption ($c^*$) to ensure contextual accuracy (e.g., describing the background or color).
2. Image-to-Image Generation
The label prompt and the selected caption are concatenated into a single weighted prompt ($p$). This text prompt, along with the original image, is fed into Stable Diffusion (v1-5).
- Prompt Weighting: The label is given a higher weight (1.50) than the caption (0.90) to prioritize the core object category over background details.
- Guidance Mapping: The CLIP similarity score dynamically determines the diffusion model's guidance scale ($g$), dictating how strictly the model must adhere to the text prompt.
- Controlled Noise: Gaussian noise is added at a controlled rate ($n$) to allow the model to alter the background, pose, or lighting while keeping the core structure intact.
Datasets
The authors evaluated SGID across seven widely recognized image classification benchmarks:
- Coarse-grained datasets: CIFAR-10, CIFAR-100, and Caltech101 (Cal101).
- Fine-grained datasets: Stanford Cars (Cars), Flowers102 (Flowers), and OxfordPets (Pets).
- Texture dataset: Describing Textures in the Wild (DTD).
Key Insights and Ideas
- The Sweet Spot of Similarity: Automated evaluations using cosine similarity show that SGID sits perfectly between rigid perturbation methods and chaotic text-to-image generators. It introduces enough variation to teach the model new features without altering the underlying class.
- Labels vs. Captions: The ablation studies revealed that while labels provide the groundtruth "anchor," captions provide crucial contextual boundaries (like background and lighting). However, for hyper-specific datasets (like Stanford Cars), captions can sometimes introduce noise, making label-only guidance more effective.
- Synergy with Traditional DA: Generative DA and perturbation DA are not mutually exclusive. Applying traditional techniques like CutMix or RandAugment on top of SGID-generated images yields compounding performance gains.
Key Results
SGID consistently outperformed standard backbones and state-of-the-art DA baselines across three distinct architectures: ResNet-50 (trained from scratch), ViT (pre-trained on ImageNet-21k), and CLIP-ViT (pre-trained on LAION-2B).
- Accuracy Gains: SGID achieved an average accuracy boost of 10.39% on ResNet-50, 2.08% on ViT, and 0.85% on CLIP-ViT across the seven datasets.
- Baseline Comparison: SGID outperformed the best generative baseline (Text2Img) by 1.72% on ResNet-50 and 0.33% on ViT.
- Hybrid Superiority: Combining SGID with RandAugment ("SGID & RA") outperformed standard RandAugment by 12.47% on ResNet-50, demonstrating a powerful mutual benefit between generative and perturbation methods.
Limitations and Future Work
Despite its success, the authors note a few limitations:
- Fine-Grained Captioning Bottleneck: Captioning models like BLIP occasionally struggle to generate accurate descriptions for highly specialized or fine-grained classes, which can slightly degrade prompt quality.
- Sensitivity to Noise Rates: The noise rate ($n$) heavily impacts downstream classification performance. Finding the optimal noise level currently requires empirical tuning, as too much noise introduces semantic corruption.
Future Directions: The authors plan to explore the application of SGID to other complex computer vision tasks, such as skin disease classification, crowd counting, and mitigating privacy attacks.
Summary updated