Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation

Jiahaoli, Yang Lu, Yuan Xie, Yanyun Qu

TL;DR

Researchers have developed the Relationship Prompt Network (RPN), a highly efficient approach to open-vocabulary semantic segmentation that ditches heavy, task-specific decoders. By using a novel "Relationship Prompt Module" to guide Vision-Language Models (VLMs) like CLIP, they achieved state-of-the-art results with only 3 million trainable parameters—just 2% of the total model size.

What Problem Does This Paper Address?

Open-vocabulary semantic segmentation (OVSS) is the "holy grail" of computer vision: the ability to segment any object in an image at the pixel level, even if the model wasn't specifically trained on that object class. Traditionally, researchers have tackled this in two ways:

  1. Two-stage methods: Use a separate network to propose masks and then use a VLM to classify them.
  2. One-stage methods: Use a complex "semantic decoder" to distill a VLM's knowledge down to the pixel level.
Comparison of the proposed prompt learning method versus existing VLM-based methods.

The problem? Both approaches are computationally expensive and rely on massive, segmentation-specific architectures. The authors of this paper asked a provocative question: Can we make the VLM do the segmentation itself using only prompts? While prompt learning is great for image-level classification, it usually fails at pixel-level tasks because it lacks the granularity to distinguish between a "cat" pixel and a "background" pixel.

Methods

The authors introduce the Relationship Prompt Network (RPN). Instead of adding a giant decoder, they "nudge" the existing VLM using clever, lightweight components:

  • Relationship Prompt Module (RPM): This is the heart of the paper. It generates prompts specifically designed to bridge the gap between text (the class name) and pixels. It uses an Image-to-Pixel (ITP) attention scheme to help the model focus on relevant pixels for a given class.
  • Multi-scale Mixture-of-Experts (M2oE): Objects come in all sizes. This module uses a gating network to choose between different "experts" (convolutional layers) to handle features at various scales, ensuring the model doesn't miss small details.
  • Adaptive Prompt Generation (APG): This allows the model to fine-tune the relationship prompt for every single pixel, enabling the high-resolution precision needed for segmentation.
  • Linear Projection Module (LPM): A tiny, two-layer linear head maps the final image and text features into a shared space to produce the final segmentation mask.

Datasets

The model was rigorously tested on four major benchmarks in both zero-shot and open-vocabulary settings:

  • Pascal VOC 2012: A classic dataset for object segmentation.
  • COCO-Stuff: A massive dataset covering 171 classes (both "things" like cars and "stuff" like sky).
  • Pascal Context: 60 classes of objects in natural scenes.
  • ADE20K: A challenging dataset with up to 847 classes, used to test the limits of open-vocabulary performance.

Key Insights and Ideas

  • VLMs already "know" pixels: The most important insight is that Vision-Language Models like CLIP already contain the spatial information needed for segmentation; it’s just buried. You don't need a new decoder to find it—you just need a better "flashlight" (the Relationship Prompt).
Layer-by-layer visualization of the model's attention heatmaps.
  • The Power of Relationships: Standard prompts are often static. By making the prompt a function of the relationship between the text and the image features, the model becomes much more sensitive to spatial context.
  • Efficiency is King: By focusing on prompt learning rather than architectural changes, the authors reduced the trainable parameters to just ~3M. This makes the model much faster to train and deploy compared to behemoths like ZegFormer.

Key Results

The RPN doesn't just match existing models; it often beats them while being significantly leaner:

  • Parameter Efficiency: It uses only 3.2M trainable parameters, compared to 13-60M in competing models like ZegCLIP or ZegFormer.
  • Speed: On an NVIDIA 1080 Ti, RPN achieves ~10 FPS, significantly faster than the 1.5–1.7 FPS of two-stage competitors.
  • Performance: In zero-shot settings on Pascal VOC, RPN achieved a mean Intersection over Union (mIoU) of 84.6%, surpassing the previous state-of-the-art (FreeSeg) by 6%.
  • Open-Vocabulary: On the massive ADE20K (A-150) benchmark, it achieved a SOTA mIoU of 31.5% using a ViT-B/16 backbone.

Limitations and Future Work

Despite its success, the authors note a few hurdles:

  • Data Dependency: The model still relies heavily on the quality of the VLM's pre-training. If the base VLM (like CLIP) hasn't seen enough diverse data, the RPN's "prompting" can only do so much.
  • Uni-modal Knowledge: The method struggles more when using weights from models trained only on images (like MAE) compared to those trained on image-text pairs.
  • Future Directions: The authors suggest that directly acting on the attention maps or dynamically reorganizing multi-head attention could make the model even more "lightweight" and precise.

In short, this paper proves that you don't need to reinvent the wheel (or the decoder) to achieve great segmentation—you just need to learn how to talk to the model you already have.

Summary updated

Abstract

Open-vocabulary semantic segmentation (OVSS) aims to segment unseen classes without corresponding labels. Existing Vision-Language Model (VLM)-based methods leverage VLM's rich knowledge to enhance additional explicit segmentation-specific networks, yielding competitive results, but at the cost of extensive training cost. To reduce the cost, we attempt to enable VLM to directly produce the segmentation results without any segmentation-specific networks. Prompt learning offers a direct and parameter-efficient approach, yet it falls short in guiding VLM for pixel-level visual classification. Therefore, we propose the ${\bf R}$elationship ${\bf P}$rompt ${\bf M}$odule (${\bf RPM}$), which generates the relationship prompt that directs VLM to extract pixel-level semantic embeddings suitable for OVSS. Moreover, RPM integrates with VLM to construct the ${\bf R}$elationship ${\bf P}$rompt ${\bf N}$etwork (${\bf RPN}$), achieving OVSS without any segmentation-specific networks. RPN attains state-of-the-art performance with merely about ${\bf 3M}$ trainable parameters (2\% of total parameters).
BibTeX
@inproceedings{jiahaoli2024relationship,
  author = {Jiahaoli and Yang Lu and Yuan Xie and Yanyun Qu},
  title = {{Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PKcCHncbzg},
}
Figure