Diffusion Models for Open-Vocabulary Segmentation

Laurynas Karazija, Iro Laina, Andrea Vedaldi, Christian Rupprecht

TL;DR

OVDiff is a training-free, annotation-free framework for open-vocabulary semantic segmentation that turns generative text-to-image diffusion models into an on-demand "factory" for image segmenters. By generating synthetic support images for any text query, it extracts visual prototypes for both foreground objects and their backgrounds, allowing it to segment target images via simple nearest-neighbor matching. Without a single weight update, OVDiff sets a new state-of-the-art on standard benchmarks, outperforming prior methods by over 5% on PASCAL VOC.

What Problem Does This Paper Address?

Open-vocabulary semantic segmentation is the holy grail of dense visual understanding: the ability to partition an image into regions corresponding to any arbitrary text description provided by a user.

While large vision-language models (like CLIP) have revolutionized image-level classification, adapting them to pixel-level localization tasks usually requires massive, expensive datasets with dense mask annotations or intensive contrastive fine-tuning. Furthermore, handling the "background"—the parts of an image that don't match any of the user's text queries—remains a major headache. Most existing methods rely on brittle, image-dependent thresholding heuristics to decide what is background, often leading to noisy and imprecise object boundaries.

OVDiff tackles these challenges by asking a clever question: Instead of training a massive new model to bridge the gap between text and pixels, can we use existing, frozen foundation models to synthesize a custom segmenter on the fly?

Methods

OVDiff operates entirely without training or fine-tuning, breaking the segmentation process down into three main steps: Generation, Representation, and Matching.

[Text Query] ➔ (Stable Diffusion) ➔ [Support Images] ➔ (Feature Extraction) ➔ [Prototypes] ➔ [Nearest-Neighbor Matching] ➔ [Final Mask]
  1. Support Set Generation: Given a textual category (e.g., "Darth Vader"), the system uses a pre-trained text-to-image model (Stable Diffusion) to generate a small support set of synthetic images ($N=32$).
  2. Representing Categories (Prototypes): To extract clean visual features from these synthetic images, OVDiff leverages the diffusion model's internal cross-attention maps to automatically separate the foreground object from its background. To capture intra-class variations and fine-grained details, it extracts three types of visual features using an ensemble of frozen networks (Stable Diffusion, DINO, and CLIP):
  3. Instance-level prototypes (from individual images).
  4. Class-level prototypes (averaged across the support set).
  5. Part-level prototypes (derived via K-Means clustering to capture sub-components of an object).

Crucially, it extracts both positive (foreground) and negative (background) prototypes.

  1. Segmentation via Prototype Matching: To segment a target image, its dense features are extracted using the same visual encoders. A simple nearest-neighbor matching scheme (using cosine similarity) compares the target pixels against the accumulated positive and negative prototypes. The pixel is assigned to whichever class prototype it matches best.
Fig. 2: OVDiff overview. Prototype sampling: text queries are used to sample a set of support images which are further processed by a feature extractor and a segmenter forming positive and negative (background) prototypes. Segmentation: image features are compared against prototypes.The CLIP filter removes irrelevant prototypes based on global image contents.

To further boost accuracy, the authors introduce Category Pre-filtering (using CLIP to narrow down which vocabulary terms are actually present in the target image) and "Stuff" Filtering (using ChatGPT to identify non-countable background classes like "sky" or "water" to prevent cross-contamination between foreground and background prototypes).

Datasets

The authors evaluated OVDiff entirely in a zero-shot/unsupervised manner on the validation splits of three major benchmarks that include realistic background settings:

  • PASCAL VOC: 20 foreground classes plus a background class.
  • PASCAL Context: A challenging dataset featuring 59 foreground classes plus background, containing a mix of distinct objects ("things") and amorphous regions ("stuff").
  • COCO-Object: A variant of COCO-Stuff containing 80 foreground object classes and a background class.

They also validated their model on background-excluded setups using ADE20K, COCO-Stuff, and Cityscapes.

Key Insights and Ideas

  • Vision-to-Vision beats Text-to-Vision: Pixel-level text-to-image alignment is notoriously noisy. By using a generative model to turn text into images first, OVDiff shifts the problem to a unimodal vision-to-vision comparison, which is significantly cleaner and more robust.
  • Exploiting Generative Contextual Priors: Generative models don't create objects in a vacuum; Stable Diffusion naturally places a "boat" on "water" under a "sky." OVDiff exploits this contextual prior by harvesting the background pixels of synthetic images to create explicit negative prototypes, elegantly solving the open-vocabulary background problem without arbitrary thresholding.
  • Granularity is Key: Relying on a single averaged class embedding fails to capture diverse appearances. Combining instance, class, and part-level clustered prototypes allows the model to effectively segment complex objects and their sub-parts.

Key Results

OVDiff achieves state-of-the-art performance among training-free open-vocabulary segmentation methods:

  • PASCAL VOC: Achieves 66.3% mIoU, outperforming the previous best training-free method (CLIP-DIY) by over 5%. When paired with a standard post-processing refinement step (PAMR), it reaches 68.4% mIoU.
  • PASCAL Context & COCO-Object: Reaches 29.7% and 34.6% mIoU respectively, consistently outperforming methods that require extensive contrastive training (like GroupViT and TCL).

The qualitative results demonstrate highly precise boundaries for both common objects and rare/fictional concepts like "Pikachu" or "Olaf from Frozen."

Fig. 1: OVDiff is an open-vocabulary segmentation method that, given an image and a free-form set of class names, can segment any user-defined classes. It is fully automatic and does not require any further training.

Limitations and Future Work

  • Resolution and "Halos": Because the method relies on the internal feature maps of frozen foundation models (like Stable Diffusion's UNet), the spatial resolution can be low. Upscaling these features sometimes creates a slight "halo" effect around segmented objects or causes the model to miss very small objects.
  • Generative Bounds: The method is strictly bounded by the capabilities of the underlying generator. If Stable Diffusion cannot accurately generate a concept (e.g., rendering legible text or highly specific medical imaging), OVDiff cannot build a segmenter for it out-of-the-box.
  • Computational Latency: While it completely bypasses upfront training costs, generating the support sets takes time (~110 seconds per new category). However, this cost is heavily amortized when processing large image collections with a fixed vocabulary.

Summary updated

Abstract

Open-vocabulary segmentation is the task of segmenting anything that can be named in an image. Recently, large-scale vision-language modelling has led to significant advances in open-vocabulary segmentation, but at the cost of gargantuan and increasing training and annotation efforts. Hence, we ask if it is possible to use existing foundation models to synthesise on-demand efficient segmentation algorithms for specific class sets, making them applicable in an open-vocabulary setting without the need to collect further data, annotations or perform training. To that end, we present OVDiff, a novel method that leverages generative text-to-image diffusion models for unsupervised open-vocabulary segmentation. OVDiff synthesises support image sets for arbitrary textual categories, creating for each a set of prototypes representative of both the category and its surrounding context (background). It relies solely on pre-trained components and outputs the synthesised segmenter directly, without training. Our approach shows strong performance on a range of benchmarks, obtaining a lead of more than 5% over prior work on PASCAL VOC.
BibTeX
@inproceedings{karazija2024diffusion,
  author = {Laurynas Karazija and Iro Laina and Andrea Vedaldi and Christian Rupprecht},
  title = {{Diffusion Models for Open-Vocabulary Segmentation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72652-1\_18},
}
Figure