DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
Lists
Only you see this
Abstract
Collecting and annotating images with pixel-wise labels is time-consuming and laborious. In contrast, synthetic data can be freely available using a generative model (e.g., DALL-E, Stable Diffusion). In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model, which uses only text-image pairs during training. Our approach, termed DiffuMask, exploits the potential of the cross-attention map between text and image, which is natural and seamless to extend the text-driven image synthesis to semantic mask generation. DiffuMask uses text-guided cross-attention information to localize class/word-specific regions, which are combined with practical techniques to create a novel high-resolution and class-discriminative pixel-wise mask. The methods help to significantly reduce data collection and annotation costs. Experiments demonstrate that the existing segmentation methods trained on synthetic data of DiffuMask can achieve a competitive performance over the counterpart of real data (VOC 2012, Cityscapes). For some classes (e.g., bird), DiffuMask presents promising performance, close to the state-of-the-art result of real data (within 3% mIoU gap). Moreover, in the open-vocabulary segmentation (zero-shot) setting, DiffuMask achieves new state-of-the-art results on the Unseen classes of VOC 2012. The project website can be found at ${\color{red}{\text{DiffuMask}}}$.
BibTeX
@inproceedings{wu2023diffumask,
author = {Weijia Wu and Yuzhong Zhao and Mike Zheng Shou and Hong Zhou and Chunhua Shen},
title = {{DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models}},
booktitle = {IEEE/CVF International Conference on Computer Vision},
year = {2023},
doi = {10.1109/iccv51070.2023.00117},
}Loading related content...