Image Augmentation Agent for Weakly Supervised Semantic Segmentation

Wangyu Wu, Xianglin Qiu, Siqi Song, Zhenhong Chen, Xiaowei Huang, Fei Ma, Jimin Xiao

TL;DR

The Image Augmentation Agent (IAA) framework shifts the focus of Weakly Supervised Semantic Segmentation (WSSS) from architecture tweaks to a data-centric approach. By combining Large Language Models (LLMs) for diverse prompt generation and Diffusion Models for high-quality image synthesis, IAA automatically expands training datasets with varied, semantically consistent images, setting new state-of-the-art performance benchmarks.

What Problem Does This Paper Address?

Semantic segmentation—the task of labeling every pixel in an image—usually requires expensive, pixel-perfect annotations. Weakly Supervised Semantic Segmentation (WSSS) tries to bypass this by training models using only image-level labels (e.g., "this image contains a cat").

While WSSS has improved significantly, most researchers focus on designing better network structures or loss functions to extract "Class Activation Maps" (CAMs). The authors of this paper argue that these methods are hitting a ceiling because they rely on fixed, limited datasets. A model can only learn so much from the same few thousand images. The challenge is: how do we create more diverse training data without the massive cost of manual labeling?

Methods

The authors propose the Image Augmentation Agent (IAA), a pipeline that uses generative AI to "hallucinate" new, useful training samples. The framework consists of three main pillars:

  1. LLM-Driven Prompt Generation with Self-Refinement (PSR): Instead of using a single fixed prompt (like "a photo of a cat"), they use GPT-4o to generate a wide variety of realistic backgrounds and contexts. To solve the "instability" of LLMs (where they might produce nonsense), they implemented a self-refinement loop: the LLM evaluates its own prompts against a quality threshold before they are used.
  2. Controlled Diffusion Generation: To ensure the new images actually look like the original objects, they use ControlNet. This allows the model to keep the structural "skeleton" of the original image (using edge or pose maps) while changing the background and style based on the LLM's prompts.
  3. Image Self-Refinement (ISR): Not every generated image is a winner. The authors integrated an "online filter"—a pre-trained Vision Transformer (ViT) classifier—that checks each generated image. If the image doesn't clearly represent the target class, it's discarded and regenerated.
Figure 2: Overall of our IAA. First, IAA uses image-level labels to generate text prompt

Datasets

The model was evaluated on two gold-standard benchmarks for semantic segmentation:

  • PASCAL VOC 2012: A dataset with 21 categories. The authors used the standard training set of 10,582 images.
  • MS COCO 2014: A much larger and more complex dataset with 81 categories and approximately 82,000 training images.

Key Insights and Ideas

  • Data Diversity > Model Complexity: The "aha moment" here is that providing a model with a "cat in a snowy field" and a "cat in a warm fireplace" helps it understand the concept of a cat better than just looking at the same "cat on a sofa" image a hundred times with different loss functions.
  • The Necessity of Self-Correction: Raw generative AI is noisy. The paper shows that simply adding diffusion-generated images actually decreased performance (from 69.3% to 69.1% mIoU) because of low-quality samples. The "Self-Refinement" mechanisms for both text and images are what actually unlock the performance gains.
  • Structural Fidelity: By using ControlNet instead of unconditioned Stable Diffusion, the agent ensures that the "weak label" (the image-level tag) remains valid for the new synthetic image.
Figure 1: (a) In the traditional WSSS pipeline, the original image is directly used for training.

Key Results

The IAA framework achieved impressive results, outperforming previous state-of-the-art (SOTA) methods:

  • PASCAL VOC 2012: Reached 72.3% mIoU, a significant jump over the previous best of 71.4%.
  • MS COCO 2014: Achieved 45.3% mIoU, setting a new benchmark for this challenging dataset.
  • Ablation Success: The Image Self-Refinement (ISR) alone improved the baseline by 2.4%, proving that quality control in synthetic data is paramount.
  • Efficiency: While the training process is more complex due to data generation, the inference time remains identical to standard models (approx. 0.95s per image), making it perfectly suitable for real-world deployment.

Limitations and Future Work

  • Training Complexity: Generating high-quality synthetic images and running LLM refinement loops adds significant computational overhead during the training phase.
  • Domain Specificity: While it works well for general objects (cats, cars, planes), its effectiveness in highly specialized domains (like medical imaging or satellite data) where LLMs might lack "world knowledge" remains to be seen.
  • Future Directions: The authors plan to extend this "augmentation agent" concept to other downstream tasks like object detection and instance segmentation, and to test its robustness in "domain-shifted" environments where the training and test data look very different.

Summary updated

Abstract

Weakly-supervised semantic segmentation (WSSS) has achieved remarkable progress using only image-level labels. However, most existing WSSS methods focus on designing new network structures and loss functions to generate more accurate dense labels, overlooking the limitations imposed by fixed datasets, which can constrain performance improvements. We argue that more diverse trainable images provides WSSS richer information and help model understand more comprehensive semantic pattern. Therefore in this paper, we introduce a novel approach called Image Augmentation Agent (IAA) which shows that it is possible to enhance WSSS from data generation perspective. IAA mainly design an augmentation agent that leverages large language models (LLMs) and diffusion models to automatically generate additional images for WSSS. In practice, to address the instability in prompt generation by LLMs, we develop a prompt self-refinement mechanism. It allow LLMs to re-evaluate the rationality of generated prompts to produce more coherent prompts. Additionally, we insert an online filter into diffusion generation process to dynamically ensure the quality and balance of generated images. Experimental results show that our method significantly surpasses state-of-the-art WSSS approaches on the PASCAL VOC 2012 and MS COCO 2014 datasets.
BibTeX
@misc{wu2024image,
  author = {Wangyu Wu and Xianglin Qiu and Siqi Song and Zhenhong Chen and Xiaowei Huang and Fei Ma and Jimin Xiao},
  title = {{Image Augmentation Agent for Weakly Supervised Semantic Segmentation}},
  year = {2024},
  eprint = {2412.20439},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20439},
}
Figure