Abstract
The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate on Few-shot Semantic Segmentation, establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.
Introduction
Few-shot Semantic Segmentation (FSS) is the challenge of identifying and segmenting a specific object category in a "query" image using only one or a few labeled "support" examples. Traditionally, this has been solved using complex, specialized architectures built on top of standard vision backbones like ResNet or DINO. Recently, the field has shifted toward In-context Segmentation, inspired by the way Large Language Models (LLMs) learn from examples in a prompt.
While generative Diffusion Models have shown incredible "priors" (internal knowledge) about image structure and semantics, adapting them for precise, discriminative tasks like segmentation is difficult. Most existing attempts either add heavy decoder heads—which can destroy the model's pre-trained knowledge—or use multi-step denoising processes that are computationally expensive and often unnecessary for simple binary masks. DiffewS addresses these challenges by repurposing pre-trained Latent Diffusion Models (LDM) with minimal structural changes, treating mask prediction as a streamlined one-step generation task.
The DiffewS Framework
The DiffewS framework is designed to maximize the use of the Latent Diffusion Model’s pre-trained weights through several key technical innovations:
KV Fusion Self-Attention (FSA)
Instead of using standard cross-attention, which often compresses information into a few tokens, DiffewS allows the query image to directly "attend" to the features of the support image. This is achieved by concatenating the Keys (K) and Values (V) of the support image with those of the query image within the UNet’s self-attention layers. This mechanism ensures that the query image can leverage the high-resolution spatial information present in the support image.
Support Mask Injection and Encoding
The support mask is converted into an RGB image, encoded into the latent space via a VAE, and concatenated with the support image latent ($z_s$). This allows the model to "see" exactly which pixels in the support image it should be matching, effectively guiding the attention mechanism toward the relevant semantic category.
One-step Image-to-Mask Generation (OI2M)
Standard diffusion involves adding noise and denoising over many steps. However, for binary masks, this process can be counterproductive because masks are structurally simple; the model can "leak" information during training if the noise doesn't sufficiently obscure the structure. DiffewS utilizes a one-step process where the UNet takes the query image and support data to directly output the predicted mask latent ($\hat{z}_{mq}$).
Supervision Format
The researchers discovered that supervising the model to produce a "white foreground on a black background" mask is more effective than trying to output masks overlaid on real images. This simplified objective helps the model focus on the discriminative task of segmentation rather than the generative task of image reconstruction.
Experimental Results
The model was evaluated on several standard benchmarks, including COCO-20i, PASCAL-5i, and LVIS-92i (for out-of-domain testing).
- In-context Superiority: In the in-context setting on COCO, DiffewS achieved a 1-shot mIoU of 71.3%, significantly outperforming the previous state-of-the-art, SegGPT (56.1%).
- Cross-Domain Performance: On the LVIS-92i dataset, it registered 31.4% in 1-shot settings, demonstrating robust generalization to rare or unseen categories.
- Efficiency: Unlike many FSS models that require days of training, DiffewS reaches peak performance in just 10,000 to 30,000 iterations (roughly 6–11 hours on a single GPU).
- Qualitative Success: The model successfully segments categories not present in its training set, such as slippers and aprons, and handles cross-style tasks, such as matching a real-world object to a cartoon version.
Discussion and Future Work
Key Insights
- Generative Priors as Discriminative Goldmines: The internal representations learned by a model trained to generate images are incredibly useful for finding objects, provided the architecture is not broken during fine-tuning.
- Minimalism Wins: By avoiding extra decoder heads and keeping the original UNet structure, DiffewS retains the "generalist" capabilities of the LDM, allowing it to handle abstract concepts that specialized models might miss.
Limitations
While the 1-shot performance is stellar, the jump to 5-shot or 10-shot isn't as large as seen in some specialized models. Additionally, the model still struggles with Appearance Disparity (when the support and query look very different), Look-alike Interference (confusing similar-looking objects), and heavy Occlusion.
Future Directions
Future work will focus on scaling DiffewS to larger datasets and expanding it to other generalist vision tasks, such as depth estimation or surface normal prediction, within the same unified framework.
Summary updated