TL;DR
SeeDiff is a training-free, off-the-shelf framework that generates pixel-accurate segmentation masks directly from Stable Diffusion without requiring pre-trained segmentation networks, extra decoders, or prompt tuning. By connecting diffusion attention layers to classical seeded segmentation, it extracts coarse object cues from cross-attention maps and progressively refines them into crisp masks using multi-scale self-attention and background inversion.
What Problem Does This Paper Address?
Training modern semantic segmentation architectures (such as Mask2Former) demands millions of laboriously annotated pixel-level masks. While synthetic data generation using text-to-image models like Stable Diffusion offers a promising alternative, generating corresponding high-quality ground-truth masks remains challenging.
Previous approaches suffer from major limitations:
- Coarse cross-attention: Cross-attention maps alone localize objects roughly, missing sharp structural boundaries.
- External dependencies & overfitting: Earlier methods often rely on pre-trained segmentation models, extra trainable decoders, or prompt tuning to clean up masks. This reintroduces supervised label bias and harms performance on novel or unseen classes.
Methods
SeeDiff draws direct inspiration from classical seeded segmentation algorithms—first locating initial seed points and then expanding them to visually and semantically similar regions.
Text Prompt ──► Cross-Attention (16x16) ──► Thresholding ──► Initial Seeds
│
┌──────────────────────── Iterative Expansion ─────────────────────┘
▼
Index Self-Attention (16x16) ──► Upsample ──► Seeds (32x32) ──► SA (32x32) ──► SA (64x64)
│
┌────────────────── Background Refinement Inversion ──────────────────────────────┘
▼
Invert Mask ──► Background Seeds ──► SA Region Expansion ──► Background Mask
│
Final Crisp Mask ◄── Subtract/Suppress Background: (1 - M_bg) ⊙ M_object
- Seed Initialization (Cross-Attention): The method extracts the cross-attention map at a balanced $16 \times 16$ resolution corresponding to the target class token. Thresholding this map provides reliable initial "seed" coordinates.
- Iterative Region Expansion (Multi-Scale Self-Attention): Because self-attention (SA) naturally models intra-image feature affinities, SeeDiff indexes the SA map using the active seed coordinates. Low-resolution SA layers propagate attention across broad semantic object parts, while higher-resolution SA layers (up to $64 \times 64$) refine boundary details. Seeds are iteratively re-extracted and upsampled at each scale.
- Mask Refinement via Background Inversion: Single-object synthetic images typically feature relatively uniform backgrounds that are easy for self-attention to correlate accurately. SeeDiff inverts the expanded object mask to extract background seeds, grows an accurate background mask via SA expansion, and suppresses background noise from the foreground mask via $(1 - M_{\text{bg}}) \odot M_{\text{fg}}$.
Datasets
- Synthetic Generation: Stable Diffusion 2-base using simple fixed prompt templates (e.g., "a photo of a <class>" and "a photo of a <class> in the city").
- Evaluation Benchmarks:
- PASCAL VOC 2012: Evaluated in standard settings as well as open-vocabulary/zero-shot splits (15 seen classes, 5 unseen classes).
- Cityscapes: Evaluated on the Human (person, rider) and Vehicle (car, bus, truck, train) categories.
- Downstream segmentation models (Mask2Former with ResNet-50 and Swin-B backbones) were trained on the synthetic datasets and evaluated against standard validation splits.
Key Insights and Ideas
- Complementary Attention Roles: Cross-attention excels at answering "where is the object mentioned in the prompt?" (localization), whereas self-attention answers "what other pixels share visual features with these points?" (grouping). Combining them mirrors classical seeded region growing.
- Higher Resolution Is Not Always Better for Cross-Attention: At $32 \times 32$ or $64 \times 64$, cross-attention tends to diffuse uniformly across the image, losing object specificity. Using a $16 \times 16$ cross-attention map strikes the best balance for seed initialization.
- Exploiting Background Uniformity: Synthetic objects often have heterogeneous textures, causing seeds to cluster on specific subparts. Conversely, synthetic backgrounds are comparatively homogeneous. Expanding background seeds and subtracting the result cleanly strips away boundary leakage.
Key Results
- PASCAL VOC 2012:
- Training Mask2Former (Swin-B) solely on 60k SeeDiff synthetic images achieved 79.6% mIoU, outperforming DiffuMask (70.6%) by 9.0% without any prompt engineering or pre-trained segmenter.
- When fine-tuned with 5.0k real images, SeeDiff reached 88.8% mIoU, substantially surpassing models trained purely on real VOC data (84.3%).
- Cityscapes: On pure synthetic training, SeeDiff achieved 74.3% IoU on Humans and 89.8% IoU on Vehicles, outperforming DiffuMask (72.1% and 87.0%).
- Zero-Shot Segmentation: Because SeeDiff avoids pre-trained segmentation networks, it maintains balanced performance across both seen and unseen classes, whereas prior baselines suffered steep drops on unseen categories.
Limitations and Future Work
- Single-Object Bias: The pipeline is designed around simple prompts with relatively clean, uniform backgrounds. Complex scenes featuring multiple interacting objects and dense clutter are not explicitly explored.
- Prompt Dependence: Generation relies on predetermined prompt templates, meaning vocabulary breadth depends entirely on the underlying diffusion model's latent representation.
- Future Work: The authors highlight SeeDiff as a completely modular, training-free mechanism, opening avenues to extend training-free seeded diffusion strategies to multi-instance detection, panoptic segmentation, and video segmentation.
Summary updated