TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation

Jingyao Li, Pengguang Chen, Shengju Qian, Jiaya Jia

Abstract

Recent success of Contrastive Language-Image Pre-training~(CLIP) has shown great promise in pixel-level open-vocabulary learning tasks. A general paradigm utilizes CLIP's text and patch embeddings to generate semantic masks. However, existing models easily misidentify input pixels from unseen classes, thus confusing novel classes with semantically-similar ones. In our work, we disentangle the ill-posed optimization problem into two parallel processes: one performs semantic matching individually, and the other judges reliability for improving discrimination ability. Motivated by special tokens in language modeling that represents sentence-level embeddings, we design a trusty token that decouples the known and novel category prediction tendency. With almost no extra overhead, we upgrade the pixel-level generalization capacity of existing models effectively. Our TagCLIP (CLIP adapting with Trusty-guidance) boosts the IoU of unseen classes by 7.4% and 1.7% on PASCAL VOC 2012 and COCO-Stuff 164K.
BibTeX
@misc{li2023tagclip,
  author = {Jingyao Li and Pengguang Chen and Shengju Qian and Jiaya Jia},
  title = {{TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation}},
  year = {2023},
  eprint = {2304.07547},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.07547},
}
Figure