TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation

Jingyao Li, Pengguang Chen, Shengju Qian, Jiaya Jia

Introduction

Zero-shot (or open-vocabulary) semantic segmentation aims to partition an image into regions belonging to categories the model has never seen during training. While modern approaches leverage the vast knowledge of CLIP (Contrastive Language-Image Pre-training), they often struggle with discrimination ability.

When a model is fine-tuned on a specific set of "seen" classes, it tends to overfit and become "too confident," forcing pixels from novel, "unseen" classes into the most similar known category. For example, a model might misidentify a playing field as dirt or clouds as sky. The authors of TagCLIP describe this as an "ill-posed optimization problem" where the model lacks a mechanism to recognize something as truly new.

Segmentation results showing the model's ability to distinguish between complex categories like playing field, dirt, and grass.

The TagCLIP Methodology

TagCLIP (Trusty-aware guided CLIP) addresses this by breaking the segmentation task into two parallel processes: Semantic Matching (identifying what a pixel resembles) and Reliability Judgment (determining if the prediction is actually trustworthy).

Key Innovations

  • The Trusty Token: A learnable token, inspired by NLP's [CLS] token, that captures "prediction tendency" to distinguish between known and novel categories.
  • Trusty Map Generation: The model produces a Trusty Map—a probability map indicating if a pixel belongs to a "seen" class. During testing, this map acts as a gate; if a pixel is likely "unseen," the map suppresses scores for seen classes, allowing novel categories to be correctly identified.
  • Streamlined Decoder: By simplifying the standard transformer-based decoder to a single-attention block structure, the authors significantly reduced computational overhead.

Results and Performance

TagCLIP sets a new State-of-the-Art (SoTA) across multiple benchmarks, demonstrating that disentangling reliability from semantics leads to both better accuracy and higher efficiency.

  • Accuracy Boost: It improved the Intersection over Union (IoU) for unseen classes by 7.4% on PASCAL VOC and 1.7% on COCO-Stuff.
  • Speed and Efficiency: The simplified architecture resulted in a ~50% increase in FPS (Frames Per Second) and a 15% reduction in parameters compared to previous leading methods like ZegCLIP.
  • Robustness: In cross-dataset scenarios, TagCLIP outperformed existing models, proving it generalizes well beyond its training data.
Example of open-vocabulary segmentation on a kite image, identifying the object against the sky.

Conclusion and Future Work

By introducing the "Trusty Token," TagCLIP provides a mechanism for models to say, "I recognize this is something new," rather than forcing a wrong fit. While the model is still limited by the underlying knowledge of the frozen CLIP backbone, it represents a significant step forward in making open-vocabulary segmentation more reliable and computationally efficient. Future research may focus on refining the semantic boundaries of the "unseen" regions identified by the trusty map.

Summary updated

Abstract

Contrastive Language-Image Pre-training (CLIP) has recently shown great promise in pixel-level zero-shot learning tasks. However, existing approaches utilizing CLIP's text and patch embeddings to generate semantic masks often misidentify input pixels from unseen classes, leading to confusion between novel classes and semantically similar ones. In this work, we propose a novel approach, TagCLIP (Trusty-aware guided CLIP), to address this issue. We disentangle the ill-posed optimization problem into two parallel processes: semantic matching performed individually and reliability judgment for improving discrimination ability. Building on the idea of special tokens in language modeling representing sentence-level embeddings, we introduce a trusty token that enables distinguishing novel classes from known ones in prediction. To evaluate our approach, we conduct experiments on two benchmark datasets, PASCAL VOC 2012, COCO-Stuff 164K and PASCAL Context. Our results show that TagCLIP improves the Intersection over Union (IoU) of unseen classes by 7.4%, 1.7% and 2.1%, respectively, with negligible overheads. The code is available at https://github.com/dvlab-research/TagCLIP.
BibTeX
@misc{li2023tagclip,
  author = {Jingyao Li and Pengguang Chen and Shengju Qian and Jiaya Jia},
  title = {{TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation}},
  year = {2023},
  eprint = {2304.07547},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.07547},
}
Figure