DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models

Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, Chunhua Shen

Abstract

Collecting and annotating images with pixel-wise labels is time-consuming and laborious. In contrast, synthetic data can be freely available using a generative model. In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the off-the-shelf Stable Diffusion model. Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image to extend text-driven image synthesis to semantic mask generation. DiffuMask uses text-guided cross-attention information to localize class-specific regions, combined with practical techniques to create high-resolution, class-discriminative pixel-wise masks. Experiments demonstrate that segmentation methods trained on DiffuMask synthetic data achieve competitive performance compared to real data (VOC 2012, Cityscapes) and set new state-of-the-art results in open-vocabulary settings.

Introduction

Semantic segmentation is a critical task in computer vision, yet it remains one of the most "data-hungry" fields due to the high cost of manual pixel-level annotation. For instance, annotating a single image in the Cityscapes dataset can take an expert up to 60 minutes. While synthetic data from GANs has been explored previously, it often requires a "seed" of labeled data to train a mask decoder. DiffuMask addresses this by extracting labels directly from the internal mechanisms of pre-trained diffusion models, effectively turning them into automatic labeling machines.

The DiffuMask Framework

The core of DiffuMask lies in the exploitation of the cross-attention mechanism inherent in text-to-image diffusion models like Stable Diffusion.

Cross-Attention Exploitation

When a diffusion model generates an image based on a prompt, it utilizes internal attention maps to correlate specific words (e.g., "bird") with spatial regions in the image. DiffuMask aggregates these maps across multiple resolutions and time steps to produce a raw probability map for the target class. This allows the model to localize objects without any explicit segmentation training.

Open-vocabulary image and semantic mask generation using cross-attention maps.

Refining the Mask

To convert raw attention maps into precise binary masks, DiffuMask employs several key techniques:

  • Adaptive Thresholding with AffinityNet: Rather than using a global threshold, the system uses AffinityNet to predict semantic relationships between adjacent pixels, allowing for image-specific binarization.
  • Noise Learning (Prune by Class): To handle imperfections in synthetic data, a "Prune by Class" strategy is used. A model is trained on initial synthetic data, and samples with low-confidence labels (the bottom 70%) are discarded to ensure a high-quality final training set.
  • Prompt Engineering: To maximize diversity, the system uses sub-class prompting (e.g., specific bird species like "Golden Bulbul") and retrieval-based prompts using CLIP to mimic real-world caption variety.

Experimental Evaluation

DiffuMask was evaluated on standard benchmarks including PASCAL VOC 2012 and Cityscapes using purely synthetic training sets.

Examples of synthetic images and their corresponding pixel-level masks generated by DiffuMask across various object categories.

Performance on Standard Benchmarks

On PASCAL VOC 2012, a model trained exclusively on DiffuMask synthetic data achieved 70.6% mIoU using a Swin-B backbone. For specific classes like "bird," the performance gap between synthetic and real human-labeled data was reduced to within 3%. The results suggest that stronger vision backbones are particularly effective at bridging the domain gap between synthetic and real textures.

Open-Vocabulary Segmentation

A significant strength of DiffuMask is its performance in zero-shot scenarios. Because Stable Diffusion is trained on a massive web-scale dataset (LAION-5B), it can generate masks for classes not present in standard segmentation datasets. DiffuMask achieved a new state-of-the-art for "Unseen" classes on VOC 2012 with 65.0% mIoU, significantly outperforming previous methods that relied on real-image supervision.

Discussion and Conclusion

DiffuMask demonstrates that the spatial information embedded in diffusion models is a powerful, "free" resource for semantic segmentation. While challenges remain—such as the computational cost of large-scale generation and the difficulty of modeling complex, multi-object scenes—the ability to generate high-quality, diverse training data without manual intervention marks a significant step forward. Future work will likely focus on closing the remaining domain gap through advanced style-mixing and improving the efficiency of the synthesis pipeline.

Summary updated

Abstract

Collecting and annotating images with pixel-wise labels is time-consuming and laborious. In contrast, synthetic data can be freely available using a generative model (e.g., DALL-E, Stable Diffusion). In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model, which uses only text-image pairs during training. Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image, which is natural and seamless to extend the text-driven image synthesis to semantic mask generation. DiffuMask uses text-guided cross-attention information to localize class/word-specific regions, which are combined with practical techniques to create a novel high-resolution and class-discriminative pixel-wise mask. The methods help to reduce data collection and annotation costs obviously. Experiments demonstrate that the existing segmentation methods trained on synthetic data of DiffuMask can achieve a competitive performance over the counterpart of real data (VOC 2012, Cityscapes). For some classes (e.g., bird), DiffuMask presents promising performance, close to the stateof-the-art result of real data (within 3% mIoU gap). Moreover, in the open-vocabulary segmentation (zero-shot) setting, DiffuMask achieves a new SOTA result on Unseen class of VOC 2012. The project website can be found at https://weijiawu.github.io/DiffusionMask/.
BibTeX
@inproceedings{wu2023diffumask,
  author = {Weijia Wu and Yuzhong Zhao and Mike Zheng Shou and Hong Zhou and Chunhua Shen},
  title = {{DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00117},
}
Figure