Abstract
Collecting and annotating images with pixel-wise labels is time-consuming and laborious. In contrast, synthetic data can be freely available using a generative model. In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the off-the-shelf Stable Diffusion model. Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image to extend text-driven image synthesis to semantic mask generation. DiffuMask uses text-guided cross-attention information to localize class-specific regions, combined with practical techniques to create high-resolution, class-discriminative pixel-wise masks. Experiments demonstrate that segmentation methods trained on DiffuMask synthetic data achieve competitive performance compared to real data (VOC 2012, Cityscapes) and set new state-of-the-art results in open-vocabulary settings.
Introduction
Semantic segmentation is a critical task in computer vision, yet it remains one of the most "data-hungry" fields due to the high cost of manual pixel-level annotation. For instance, annotating a single image in the Cityscapes dataset can take an expert up to 60 minutes. While synthetic data from GANs has been explored previously, it often requires a "seed" of labeled data to train a mask decoder. DiffuMask addresses this by extracting labels directly from the internal mechanisms of pre-trained diffusion models, effectively turning them into automatic labeling machines.
The DiffuMask Framework
The core of DiffuMask lies in the exploitation of the cross-attention mechanism inherent in text-to-image diffusion models like Stable Diffusion.
Cross-Attention Exploitation
When a diffusion model generates an image based on a prompt, it utilizes internal attention maps to correlate specific words (e.g., "bird") with spatial regions in the image. DiffuMask aggregates these maps across multiple resolutions and time steps to produce a raw probability map for the target class. This allows the model to localize objects without any explicit segmentation training.
Refining the Mask
To convert raw attention maps into precise binary masks, DiffuMask employs several key techniques:
- Adaptive Thresholding with AffinityNet: Rather than using a global threshold, the system uses AffinityNet to predict semantic relationships between adjacent pixels, allowing for image-specific binarization.
- Noise Learning (Prune by Class): To handle imperfections in synthetic data, a "Prune by Class" strategy is used. A model is trained on initial synthetic data, and samples with low-confidence labels (the bottom 70%) are discarded to ensure a high-quality final training set.
- Prompt Engineering: To maximize diversity, the system uses sub-class prompting (e.g., specific bird species like "Golden Bulbul") and retrieval-based prompts using CLIP to mimic real-world caption variety.
Experimental Evaluation
DiffuMask was evaluated on standard benchmarks including PASCAL VOC 2012 and Cityscapes using purely synthetic training sets.
Performance on Standard Benchmarks
On PASCAL VOC 2012, a model trained exclusively on DiffuMask synthetic data achieved 70.6% mIoU using a Swin-B backbone. For specific classes like "bird," the performance gap between synthetic and real human-labeled data was reduced to within 3%. The results suggest that stronger vision backbones are particularly effective at bridging the domain gap between synthetic and real textures.
Open-Vocabulary Segmentation
A significant strength of DiffuMask is its performance in zero-shot scenarios. Because Stable Diffusion is trained on a massive web-scale dataset (LAION-5B), it can generate masks for classes not present in standard segmentation datasets. DiffuMask achieved a new state-of-the-art for "Unseen" classes on VOC 2012 with 65.0% mIoU, significantly outperforming previous methods that relied on real-image supervision.
Discussion and Conclusion
DiffuMask demonstrates that the spatial information embedded in diffusion models is a powerful, "free" resource for semantic segmentation. While challenges remain—such as the computational cost of large-scale generation and the difficulty of modeling complex, multi-object scenes—the ability to generate high-quality, diverse training data without manual intervention marks a significant step forward. Future work will likely focus on closing the remaining domain gap through advanced style-mixing and improving the efficiency of the synthesis pipeline.
Summary updated