Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification

Bohan Li, Xiao Xu, Xinghao Wang, Yutai Hou, Yunlong Feng, Feng Wang, Xuanliang Zhang, Qingfu Zhu, Wanxiang Che

TL;DR

The paper introduces SGID (Semantic-guided Generative Image Augmentation), a novel data augmentation method that leverages diffusion models to generate highly diverse training images while strictly preserving their original semantic meaning. By combining groundtruth labels and AI-generated captions as textual guidance, SGID strikes an optimal balance between image diversity and semantic consistency, leading to significant accuracy boosts across various vision backbones and datasets.

What Problem Does This Paper Address?

Data augmentation (DA) is essential for training deep neural networks, especially when labeled data is scarce. Traditional DA techniques generally fall into two camps, both with distinct bottlenecks:

  1. Perturbation-based methods (e.g., CutMix, Random Erasing) modify local regions of an image. While they preserve the original semantics perfectly, they lack visual diversity.
  2. Generative methods (e.g., Text2Img via Diffusion Models) synthesize entirely new images. They offer incredible diversity but frequently suffer from semantic drift—altering crucial details like adding an extra door handle to a car or changing its brand badge, which misleads the image classifier during training.

SGID addresses this trade-off by introducing a framework that injects explicit semantic guidance into an image-to-image diffusion pipeline, ensuring the generated images are both highly diverse and semantically accurate.

Methods

SGID splits the generative augmentation process into two main steps:

Figure 2: An illustration of our SGID. Step 1 first collects textual labels for each image, then use BLIP to generate captions, and then use CLIP to calculate the similarity between the chosen caption and the original image. Step 2 generates the augmented images through the Stable Diffusion model, utilizing the original image, the prompt consisting of the textual label and caption, the noise rate, and the guidance scale based on the similarity.

1. Label Acquisition and Caption Generation

To capture the exact semantics of an original image $x$, SGID combines its groundtruth class label with a descriptive caption:

  • Label Prompt: The class label is converted into a simple sentence (e.g., "A picture of a [Chevrolet Silverado]").
  • Caption Generation: The BLIP model generates a pool of candidate captions for the image using nucleus sampling or beam search.
  • CLIP Filtering: A pre-trained CLIP model ranks these captions based on their cosine similarity to the original image, selecting the highest-scoring caption ($c^*$) to ensure contextual accuracy (e.g., describing the background or color).

2. Image-to-Image Generation

The label prompt and the selected caption are concatenated into a single weighted prompt ($p$). This text prompt, along with the original image, is fed into Stable Diffusion (v1-5).

  • Prompt Weighting: The label is given a higher weight (1.50) than the caption (0.90) to prioritize the core object category over background details.
  • Guidance Mapping: The CLIP similarity score dynamically determines the diffusion model's guidance scale ($g$), dictating how strictly the model must adhere to the text prompt.
  • Controlled Noise: Gaussian noise is added at a controlled rate ($n$) to allow the model to alter the background, pose, or lighting while keeping the core structure intact.

Datasets

The authors evaluated SGID across seven widely recognized image classification benchmarks:

  • Coarse-grained datasets: CIFAR-10, CIFAR-100, and Caltech101 (Cal101).
  • Fine-grained datasets: Stanford Cars (Cars), Flowers102 (Flowers), and OxfordPets (Pets).
  • Texture dataset: Describing Textures in the Wild (DTD).

Key Insights and Ideas

  • The Sweet Spot of Similarity: Automated evaluations using cosine similarity show that SGID sits perfectly between rigid perturbation methods and chaotic text-to-image generators. It introduces enough variation to teach the model new features without altering the underlying class.
  • Labels vs. Captions: The ablation studies revealed that while labels provide the groundtruth "anchor," captions provide crucial contextual boundaries (like background and lighting). However, for hyper-specific datasets (like Stanford Cars), captions can sometimes introduce noise, making label-only guidance more effective.
  • Synergy with Traditional DA: Generative DA and perturbation DA are not mutually exclusive. Applying traditional techniques like CutMix or RandAugment on top of SGID-generated images yields compounding performance gains.

Key Results

SGID consistently outperformed standard backbones and state-of-the-art DA baselines across three distinct architectures: ResNet-50 (trained from scratch), ViT (pre-trained on ImageNet-21k), and CLIP-ViT (pre-trained on LAION-2B).

SGID achieves the highest average performance improvement compared to other semantic-heavy or diversity-heavy baselines.
  • Accuracy Gains: SGID achieved an average accuracy boost of 10.39% on ResNet-50, 2.08% on ViT, and 0.85% on CLIP-ViT across the seven datasets.
  • Baseline Comparison: SGID outperformed the best generative baseline (Text2Img) by 1.72% on ResNet-50 and 0.33% on ViT.
  • Hybrid Superiority: Combining SGID with RandAugment ("SGID & RA") outperformed standard RandAugment by 12.47% on ResNet-50, demonstrating a powerful mutual benefit between generative and perturbation methods.

Limitations and Future Work

Despite its success, the authors note a few limitations:

  • Fine-Grained Captioning Bottleneck: Captioning models like BLIP occasionally struggle to generate accurate descriptions for highly specialized or fine-grained classes, which can slightly degrade prompt quality.
  • Sensitivity to Noise Rates: The noise rate ($n$) heavily impacts downstream classification performance. Finding the optimal noise level currently requires empirical tuning, as too much noise introduces semantic corruption.

Future Directions: The authors plan to explore the application of SGID to other complex computer vision tasks, such as skin disease classification, crowd counting, and mitigating privacy attacks.

Summary updated

Abstract

Existing image augmentation methods consist of two categories: perturbation-based methods and generative methods. Perturbation-based methods apply pre-defined perturbations to augment an original image, but only locally vary the image, thus lacking image diversity. In contrast, generative methods bring more image diversity in the augmented images but may not preserve semantic consistency, thus incorrectly changing the essential semantics of the original image. To balance image diversity and semantic consistency in augmented images, we propose SGID, a Semantic-guided Generative Image augmentation method with Diffusion models for image classification. Specifically, SGID employs diffusion models to generate augmented images with good image diversity. More importantly, SGID takes image labels and captions as guidance to maintain semantic consistency between the augmented and original images. Experimental results show that SGID outperforms the best augmentation baseline by 1.72% on ResNet-50 (from scratch), 0.33% on ViT (ImageNet-21k), and 0.14% on CLIP-ViT (LAION-2B). Moreover, SGID can be combined with other image augmentation baselines and further improves the overall performance. We demonstrate the semantic consistency and image diversity of SGID through quantitative human and automated evaluations, as well as qualitative case studies.
BibTeX
@inproceedings{li2024semantic,
  author = {Bohan Li and Xiao Xu and Xinghao Wang and Yutai Hou and Yunlong Feng and Feng Wang and Xuanliang Zhang and Qingfu Zhu and Wanxiang Che},
  title = {{Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28084},
}
Figure