CAT-Seg: Cost Aggregation for Open-vocabulary Semantic Segmentation

Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, Seungryong KimSubmission to ICLR 2024 · Withdrawn

Abstract

Existing works on open-vocabulary semantic segmentation have utilized large-scale vision-language models, such as CLIP, to leverage their exceptional open-vocabulary recognition capabilities. However, the problem of transferring these capabilities learned from image-level supervision to the pixel-level task of segmentation and addressing arbitrary unseen categories at inference makes this task challenging. To address these issues, we aim to attentively relate objects within an image to given categories by leveraging relational information among class categories and visual semantics through aggregation, while also adapting the CLIP representations to the pixel-level task. However, we observe that direct optimization of the CLIP embeddings can harm its open-vocabulary capabilities. In this regard, we propose an alternative approach to optimize the image-text similarity map, i.e. the cost map, using a novel cost aggregation-based method. Our framework, namely CAT-Seg, achieves state-of-the-art performance across all benchmarks.

Introduction

Semantic segmentation—assigning a class label to every pixel in an image—is traditionally limited to a fixed set of categories (e.g., "car," "tree," "road"). Open-vocabulary semantic segmentation aims to break these limits, allowing a model to segment an image based on any arbitrary text description.

To achieve this, researchers typically lean on vision-language foundation models like CLIP. However, there is a major hurdle: CLIP is trained on image-level data, not pixel-level data. When researchers try to fine-tune CLIP for the "dense" task of segmentation, the model tends to overfit aggressively to the specific classes seen during training, losing its ability to recognize "unseen" classes. Consequently, most existing methods freeze the CLIP encoders, which limits the model's ultimate performance.

Methodology: The CAT-Seg Framework

The core innovation of CAT-Seg (Cost Aggregation for Open-Vocabulary Semantic Segmentation) is shifting the focus from feature aggregation to cost aggregation. Instead of feeding raw image features into a decoder, the model computes the cosine similarity between every pixel embedding and every text embedding to create a "cost volume."

  1. Cost Volume Construction: This volume acts as a set of rough, noisy masks for every candidate class.
  2. Spatial & Class Aggregation: A Swin Transformer refines the masks by looking at neighboring pixels (spatial smoothness), while a Linear Transformer reasons across different classes to resolve ambiguities (e.g., distinguishing a chair from a sofa).
  3. Embedding Guidance: To prevent the cost volume from losing context, raw CLIP embeddings are injected back into the aggregation layers as guidance.
  4. Efficient Fine-tuning: The authors discovered that fine-tuning only the Query and Value (QV) projections in the CLIP encoders provides the best balance of performance and efficiency, preventing the "representation collapse" that usually occurs during full fine-tuning.
Qualitative comparison of similarity maps between standard CLIP, fine-tuned CLIP, and the proposed Aggregated Cost method. The cost aggregation approach produces significantly cleaner and more accurate object localization.

Experimental Results

CAT-Seg was evaluated on standard benchmarks (ADE20K, PASCAL VOC, PASCAL Context) and the MESS benchmark, which includes 22 diverse datasets spanning medical imaging, satellite imagery, and biology to test out-of-distribution generalization.

Performance on Seen and Unseen Classes

A key insight is that while features are class-specific, matching costs are task-specific. By training the model to refine similarity scores rather than raw embeddings, it learns the logic of matching, which generalizes far better to classes it has never seen before. Qualitative results demonstrate that while feature-based models often fail to identify unseen classes, CAT-Seg identifies them clearly because it has learned how to process the similarity between arbitrary text and image pixels.

Comparison of mIoU for seen and unseen classes on ADE20K. Cost aggregation (purple) significantly outperforms feature aggregation (blue), particularly in the fine-tuned setting for unseen categories.

State-of-the-Art and Efficiency

CAT-Seg achieved significant margins over previous bests, notably gaining +3.6 mIoU on ADE20K-847 and +8.1 mIoU on PASCAL Context-459. Furthermore, as a "one-stage" model, it is roughly 3.7x faster for inference than traditional region-based competitors that rely on mask proposals.

Conclusion

CAT-Seg proves that with the right architecture (cost aggregation) and a surgical approach to parameter updates (QV fine-tuning), fine-tuning CLIP is a massive performance multiplier for semantic segmentation. While the model generalizes well, it is still limited by the base CLIP model's knowledge in highly specialized fields like medical science. Future work may focus on addressing ground-truth ambiguity in benchmarks and improving resolution handling for Vision Transformer architectures.

Summary updated

Abstract

In this paper, we reinterpret the challenge of open-vocabulary semantic segmentation, where each pixel in an image is labeled with a wide range of text descriptions, as a correspondence problem focusing on the optimal text matching for each pixel. Addressing the limitations of conventional region-to-text matching approaches, we introduce a novel framework, CAT-Seg, grounded on the principles of cost aggregation methods in visual correspondence tasks. This framework refines the initial matching scores between dense image and text embeddings, leveraging a Transformer-based module for cost aggregation, further enhanced with embedding guidance. Notably, by operating on cosine similarity instead of manipulating embeddings directly, our approach enables the end-to-end fine-tuning of the CLIP model for pixel-level tasks, while yielding superior zero-shot capabilities. Empirical evaluations show our method's superior performance, achieving state-of-the-art results across open-vocabulary benchmarks, practical computational efficiency, and robustness for various domains, underscoring its potential for a wide range of open-vocabulary semantic segmentation applications.
BibTeX
@misc{cho2023cat,
  author = {Seokju Cho and Heeseong Shin and Sunghwan Hong and Anurag Arnab and Paul Hongsuck Seo and Seungryong Kim},
  title = {{CAT-Seg: Cost Aggregation for Open-vocabulary Semantic Segmentation}},
  year = {2023},
  eprint = {2303.11797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.11797},
}
Figure