Abstract
Semantic segmentation networks typically require extensive pixel-level annotations, a process that is both labor-intensive and costly. While recent research has attempted to leverage the internal representations of text-to-image diffusion models to generate synthetic training data, these methods often produce coarse masks or require auxiliary segmentation networks and extensive prompt tuning. We present SeeDiff, a training-free, "off-the-shelf" method that generates high-quality semantic segmentation masks directly from Stable Diffusion. By drawing a parallel between diffusion attention mechanisms and classical seeded segmentation, SeeDiff utilizes cross-attention maps as initial seeds and self-attention maps for iterative region expansion. Our approach achieves state-of-the-art performance on benchmarks like Pascal VOC 2012 without additional training or manual labeling.
Introduction
The "data-hungry" nature of semantic segmentation remains a significant bottleneck in computer vision. To recognize objects at the pixel level, models like Mask2Former require millions of images paired with precise, hand-drawn masks. While generative models like Stable Diffusion can synthesize realistic images from text prompts, extracting accurate segmentation masks from these models is non-trivial.
Previous attempts to bridge this gap often relied on Cross-Attention (CA) maps, which link text tokens to specific image regions. However, CA maps are inherently blurry and lack the sharp boundaries required for high-quality segmentation. Other methods introduced "ad-hoc" decoders or required pre-trained segmentation models to refine the output, which introduces new dependencies and limits the model's ability to generalize to unseen classes. SeeDiff addresses these issues by exploiting the inherent structural knowledge within the diffusion model's attention layers.
Methodology: From Seeds to Masks
SeeDiff is inspired by classical "region growing" algorithms from the 1990s. The framework operates through three primary stages: seed initialization, iterative expansion, and background refinement.
1. Seed Initialization (Cross-Attention)
The process begins with Cross-Attention. When a prompt such as "a photo of a dog" is processed, the model's CA maps highlight the regions most relevant to the "dog" token. While these maps are too coarse to serve as final masks, they provide highly reliable "seeds"—initial pixels that almost certainly belong to the target object.
2. Iterative Region Expansion (Self-Attention)
To transform these seeds into a precise mask, SeeDiff leverages Self-Attention (SA). SA maps quantify the semantic correspondence between every pixel in an image. The core logic is straightforward: if a seed pixel belongs to a "dog," other pixels that share high self-attention with that seed are likely part of the same object.
SeeDiff performs this expansion across multiple scales (from 16x16 to 64x64 resolution). As the resolution increases, the model iteratively refines the mask, using the semantic structure captured in the SA maps to fill in fine-grained details like limbs, spokes, or thin edges that CA maps typically miss.
3. Background Refinement
Synthetic images generated by diffusion models often feature relatively uniform backgrounds. SeeDiff turns this characteristic into an advantage. By inverting the initial foreground mask, the model identifies "background seeds." It then expands these seeds to create a dedicated background mask, which is subtracted from the foreground. This dual-masking approach significantly sharpens object boundaries and reduces noise in the final output.
Experimental Results
We evaluated SeeDiff by generating synthetic datasets to train a standard Mask2Former architecture. The method was tested across several rigorous benchmarks:
- Pascal VOC 2012: SeeDiff achieved a 79.6% mIoU (Mean Intersection over Union), a substantial improvement over the previous state-of-the-art baseline, DiffuMask (70.6%).
- Cityscapes: The model demonstrated strong performance in urban environments, specifically for "Human" and "Vehicle" categories.
- Zero-Shot/Open-Vocabulary: Because SeeDiff does not rely on a pre-trained segmentation network limited to specific classes, it generalizes effectively to rare or unseen categories such as "mermaid" or "sculpture."
Key Insights
- Complementary Attention: The "Aha!" moment of this research is the realization that CA and SA play complementary roles. CA provides the location (the seed), while SA provides the structure (the expansion).
- Structural Integrity: While CA maps become noisier at higher resolutions, SA maps maintain semantic correspondence, making them superior for capturing complex object geometries.
- Efficiency: SeeDiff is truly "off-the-shelf." It requires no additional training, no fine-tuning of the diffusion model, and no complex prompt engineering.
Limitations and Future Work
Currently, the internal self-attention maps of Stable Diffusion are limited to a maximum resolution of 64x64. While SeeDiff employs upsampling to produce 512x512 masks, native high-resolution attention would likely improve results further. Additionally, while the method excels in simple or single-object scenes, extremely cluttered environments with overlapping objects of the same class remain a challenge for the current region expansion logic. Future work will explore integrating SeeDiff with larger foundation models and more sophisticated background modeling to further automate dataset generation.
Summary updated