Abstract
Existing works on open-vocabulary semantic segmentation have utilized large-scale vision-language models, such as CLIP, to leverage their exceptional open-vocabulary recognition capabilities. However, the problem of transferring these capabilities learned from image-level supervision to the pixel-level task of segmentation and addressing arbitrary unseen categories at inference makes this task challenging. To address these issues, we aim to attentively relate objects within an image to given categories by leveraging relational information among class categories and visual semantics through aggregation, while also adapting the CLIP representations to the pixel-level task. However, we observe that direct optimization of the CLIP embeddings can harm its open-vocabulary capabilities. In this regard, we propose an alternative approach to optimize the image-text similarity map, i.e. the cost map, using a novel cost aggregation-based method. Our framework, namely CAT-Seg, achieves state-of-the-art performance across all benchmarks.
Introduction
Semantic segmentation—assigning a class label to every pixel in an image—is traditionally limited to a fixed set of categories (e.g., "car," "tree," "road"). Open-vocabulary semantic segmentation aims to break these limits, allowing a model to segment an image based on any arbitrary text description.
To achieve this, researchers typically lean on vision-language foundation models like CLIP. However, there is a major hurdle: CLIP is trained on image-level data, not pixel-level data. When researchers try to fine-tune CLIP for the "dense" task of segmentation, the model tends to overfit aggressively to the specific classes seen during training, losing its ability to recognize "unseen" classes. Consequently, most existing methods freeze the CLIP encoders, which limits the model's ultimate performance.
Methodology: The CAT-Seg Framework
The core innovation of CAT-Seg (Cost Aggregation for Open-Vocabulary Semantic Segmentation) is shifting the focus from feature aggregation to cost aggregation. Instead of feeding raw image features into a decoder, the model computes the cosine similarity between every pixel embedding and every text embedding to create a "cost volume."
- Cost Volume Construction: This volume acts as a set of rough, noisy masks for every candidate class.
- Spatial & Class Aggregation: A Swin Transformer refines the masks by looking at neighboring pixels (spatial smoothness), while a Linear Transformer reasons across different classes to resolve ambiguities (e.g., distinguishing a chair from a sofa).
- Embedding Guidance: To prevent the cost volume from losing context, raw CLIP embeddings are injected back into the aggregation layers as guidance.
- Efficient Fine-tuning: The authors discovered that fine-tuning only the Query and Value (QV) projections in the CLIP encoders provides the best balance of performance and efficiency, preventing the "representation collapse" that usually occurs during full fine-tuning.
Experimental Results
CAT-Seg was evaluated on standard benchmarks (ADE20K, PASCAL VOC, PASCAL Context) and the MESS benchmark, which includes 22 diverse datasets spanning medical imaging, satellite imagery, and biology to test out-of-distribution generalization.
Performance on Seen and Unseen Classes
A key insight is that while features are class-specific, matching costs are task-specific. By training the model to refine similarity scores rather than raw embeddings, it learns the logic of matching, which generalizes far better to classes it has never seen before. Qualitative results demonstrate that while feature-based models often fail to identify unseen classes, CAT-Seg identifies them clearly because it has learned how to process the similarity between arbitrary text and image pixels.
State-of-the-Art and Efficiency
CAT-Seg achieved significant margins over previous bests, notably gaining +3.6 mIoU on ADE20K-847 and +8.1 mIoU on PASCAL Context-459. Furthermore, as a "one-stage" model, it is roughly 3.7x faster for inference than traditional region-based competitors that rely on mask proposals.
Conclusion
CAT-Seg proves that with the right architecture (cost aggregation) and a surgical approach to parameter updates (QV fine-tuning), fine-tuning CLIP is a massive performance multiplier for semantic segmentation. While the model generalizes well, it is still limited by the base CLIP model's knowledge in highly specialized fields like medical science. Future work may focus on addressing ground-truth ambiguity in benchmarks and improving resolution handling for Vision Transformer architectures.
Summary updated