Abstract
This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group tokens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one v.s. one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes are estimated from source features to serve as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we propose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts at different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-of-the-art performance on several benchmark datasets.
Introduction
Semantic segmentation typically requires dense, pixel-perfect annotations, which are prohibitively expensive to scale. Open-Vocabulary Semantic Segmentation (OVSS) aims to overcome this by segmenting objects of any class, including those unseen during training. A more challenging variant is Weakly Open-Vocabulary Semantic Segmentation (WOVSS), where models are trained solely on image-text pairs (e.g., captions) without any mask-level supervision.
Current WOVSS methods utilize "group tokens" to cluster image pixels into semantic regions. However, these models suffer from granularity inconsistency. During training, group tokens are typically averaged to match a single text embedding (all-to-one), but during inference, each token is expected to independently identify a specific object (one-to-one). This lack of individual supervision leads to segments that are either redundant or semantically incomplete.
The PGSeg Framework
To address these issues, we introduce the Prototypical Guidance Segmentation Network (PGSeg). The core innovation is Non-learnable Prototypical Regularization (NPR), which provides explicit supervision for individual group tokens by mining "prototypes" directly from the data.
Non-learnable Prototypical Regularization (NPR)
Instead of relying on learnable parameters that may drift, NPR uses an Expectation-Maximization (EM) algorithm with Gaussian Mixture Models to extract high-quality pseudo-targets (prototypes) from image and text features. These prototypes serve as anchors for a contrastive loss, forcing each group token to align with a distinct, meaningful semantic region.
Multi-modal Guidance
PGSeg leverages two distinct types of prototypical sources:
- I-NPR (Image-level): Uses low-level image features to ensure segments are compact, following natural object boundaries and reducing redundancy.
- T-NPR (Text-level): Uses textual embeddings from the caption to ensure segments are rich, capturing the full semantic extent of the objects described.
These components are integrated into "PG Units" inserted at multiple levels of a Vision Transformer (ViT) backbone, allowing the model to progressively refine segmentation from coarse regions to detailed shapes.
Experimental Results
We evaluated PGSeg on several standard benchmarks, including PASCAL VOC12, COCO Object, and the long-tail LVIS dataset. The model was trained on large-scale datasets like CC12M and RedCaps.
- State-of-the-Art Performance: PGSeg achieved 53.2% mIoU on PASCAL VOC12 and 28.7% mIoU on COCO, significantly outperforming previous leaders such as GroupViT and ViewCo by margins of up to 5.8%.
- Efficiency: Despite the inclusion of the EM algorithm, PGSeg is computationally efficient. By reducing the number of tokens processed in deeper layers through grouping, it maintains a lower FLOP count (13.6G) compared to a standard ViT-S (16.7G).
- Semantic Integrity: Qualitative comparisons show that while models like Meta’s Segment Anything Model (SAM) often over-segment objects into parts, PGSeg is superior at capturing "semantic wholes," such as entire backgrounds or complete complex objects.
Conclusion
PGSeg demonstrates that explicit prototypical supervision can effectively bridge the granularity gap in weakly supervised segmentation. By balancing compactness and richness through multi-modal prototypes, the model achieves robust open-vocabulary performance without the need for expensive manual masks. Future work will focus on improving alignment for rare objects in massive vocabularies and integrating these prototypical patterns into larger foundation models.
Summary updated