SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models

Joon Hyun Park, Kumju Jo, Sungyong Baik

Abstract

Semantic segmentation networks typically require extensive pixel-level annotations, a process that is both labor-intensive and costly. While recent research has attempted to leverage the internal representations of text-to-image diffusion models to generate synthetic training data, these methods often produce coarse masks or require auxiliary segmentation networks and extensive prompt tuning. We present SeeDiff, a training-free, "off-the-shelf" method that generates high-quality semantic segmentation masks directly from Stable Diffusion. By drawing a parallel between diffusion attention mechanisms and classical seeded segmentation, SeeDiff utilizes cross-attention maps as initial seeds and self-attention maps for iterative region expansion. Our approach achieves state-of-the-art performance on benchmarks like Pascal VOC 2012 without additional training or manual labeling.

Introduction

The "data-hungry" nature of semantic segmentation remains a significant bottleneck in computer vision. To recognize objects at the pixel level, models like Mask2Former require millions of images paired with precise, hand-drawn masks. While generative models like Stable Diffusion can synthesize realistic images from text prompts, extracting accurate segmentation masks from these models is non-trivial.

Previous attempts to bridge this gap often relied on Cross-Attention (CA) maps, which link text tokens to specific image regions. However, CA maps are inherently blurry and lack the sharp boundaries required for high-quality segmentation. Other methods introduced "ad-hoc" decoders or required pre-trained segmentation models to refine the output, which introduces new dependencies and limits the model's ability to generalize to unseen classes. SeeDiff addresses these issues by exploiting the inherent structural knowledge within the diffusion model's attention layers.

Methodology: From Seeds to Masks

SeeDiff is inspired by classical "region growing" algorithms from the 1990s. The framework operates through three primary stages: seed initialization, iterative expansion, and background refinement.

1. Seed Initialization (Cross-Attention)

The process begins with Cross-Attention. When a prompt such as "a photo of a dog" is processed, the model's CA maps highlight the regions most relevant to the "dog" token. While these maps are too coarse to serve as final masks, they provide highly reliable "seeds"—initial pixels that almost certainly belong to the target object.

2. Iterative Region Expansion (Self-Attention)

To transform these seeds into a precise mask, SeeDiff leverages Self-Attention (SA). SA maps quantify the semantic correspondence between every pixel in an image. The core logic is straightforward: if a seed pixel belongs to a "dog," other pixels that share high self-attention with that seed are likely part of the same object.

SeeDiff performs this expansion across multiple scales (from 16x16 to 64x64 resolution). As the resolution increases, the model iteratively refines the mask, using the semantic structure captured in the SA maps to fill in fine-grained details like limbs, spokes, or thin edges that CA maps typically miss.

3. Background Refinement

Synthetic images generated by diffusion models often feature relatively uniform backgrounds. SeeDiff turns this characteristic into an advantage. By inverting the initial foreground mask, the model identifies "background seeds." It then expands these seeds to create a dedicated background mask, which is subtracted from the foreground. This dual-masking approach significantly sharpens object boundaries and reduces noise in the final output.

Experimental Results

We evaluated SeeDiff by generating synthetic datasets to train a standard Mask2Former architecture. The method was tested across several rigorous benchmarks:

  • Pascal VOC 2012: SeeDiff achieved a 79.6% mIoU (Mean Intersection over Union), a substantial improvement over the previous state-of-the-art baseline, DiffuMask (70.6%).
  • Cityscapes: The model demonstrated strong performance in urban environments, specifically for "Human" and "Vehicle" categories.
  • Zero-Shot/Open-Vocabulary: Because SeeDiff does not rely on a pre-trained segmentation network limited to specific classes, it generalizes effectively to rare or unseen categories such as "mermaid" or "sculpture."

Key Insights

  • Complementary Attention: The "Aha!" moment of this research is the realization that CA and SA play complementary roles. CA provides the location (the seed), while SA provides the structure (the expansion).
  • Structural Integrity: While CA maps become noisier at higher resolutions, SA maps maintain semantic correspondence, making them superior for capturing complex object geometries.
  • Efficiency: SeeDiff is truly "off-the-shelf." It requires no additional training, no fine-tuning of the diffusion model, and no complex prompt engineering.

Limitations and Future Work

Currently, the internal self-attention maps of Stable Diffusion are limited to a maximum resolution of 64x64. While SeeDiff employs upsampling to produce 512x512 masks, native high-resolution attention would likely improve results further. Additionally, while the method excels in simple or single-object scenes, extremely cluttered environments with overlapping objects of the same class remain a challenge for the current region expansion logic. Future work will explore integrating SeeDiff with larger foundation models and more sophisticated background modeling to further automate dataset generation.

Summary updated

Abstract

Entrusted with the goal of pixel-level object classification, the semantic segmentation networks entail the laborious preparation of pixel-level annotation masks. To obtain pixel-level annotation masks for a given class without human efforts, recent few works have proposed to generate pairs of images and annotation masks by employing image and text relationships modeled by text-to-image generative models, especially Stable Diffusion. However, these works do not fully exploit the capability of text-guided Diffusion models and thus require a pre-trained segmentation network, careful text prompt tuning, or the training of a segmentation network to generate final annotation masks. In this work, we take a closer look at attention mechanisms of Stable Diffusion, from which we draw connections with classical seeded segmentation approaches. In particular, we show that cross-attention alone provides very coarse object localization, which however can provide initial seeds. Then, akin to region expansion in seeded segmentation, we utilize the semantic-correspondence-modeling capability of self-attention to iteratively spread the attention to the whole class from the seeds using multi-scale self-attention maps. We also observe that a simple-text-guided synthetic image often has a uniform background, which is easier to find correspondences, compared to complex-structured objects. Thus, we further refine a mask using a more accurate background mask. Our proposed method, dubbed SeeDiff, generates high-quality masks off-the-shelf from Stable Diffusion, without additional training procedure, prompt tuning, or a pre-trained segmentation network.
BibTeX
@inproceedings{park2025seediff,
  author = {Joon Hyun Park and Kumju Jo and Sungyong Baik},
  title = {{SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=KgFBoWtYRd},
}
Figure