Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, Liang-Chieh Chen

Abstract

Open-vocabulary segmentation (OVS) is a challenging task requiring the segmentation and recognition of objects from an open set of categories. To bridge the gap between closed-vocabulary and open-vocabulary recognition, existing methods typically leverage multi-modal models like CLIP. However, most current approaches adopt a two-stage framework: a mask generator first proposes regions, which are then fed into a CLIP model for classification. This process is computationally inefficient, requiring multiple feature extraction passes. In this paper, we propose FC-CLIP, a single-stage framework built on a shared Frozen Convolutional CLIP backbone. FC-CLIP simplifies the pipeline while achieving a superior accuracy-cost trade-off. Our observations indicate that a frozen convolutional CLIP backbone not only maintains open-vocabulary classification capabilities but also serves as a robust mask generator that generalizes better to high resolutions than Vision Transformer (ViT) alternatives. When trained only on COCO panoptic data, FC-CLIP achieves state-of-the-art zero-shot performance across ADE20K, Mapillary Vistas, and Cityscapes, while being significantly faster and more parameter-efficient than prior art.

Introduction

Open-vocabulary segmentation (OVS) represents a significant leap toward universal scene understanding, aiming to recognize and segment any object, including those not encountered during training. The primary catalyst for progress in this field has been Contrastive Language-Image Pre-training (CLIP), which aligns image and text features in a shared embedding space.

Despite the power of CLIP, integrating it into dense prediction tasks like segmentation has historically led to complex, two-stage architectures. These systems typically use a dedicated mask generator (e.g., Mask2Former) to identify regions and then crop those regions to be processed by a separate CLIP encoder. This redundancy results in high latency and massive parameter counts. FC-CLIP addresses these inefficiencies by unifying the pipeline into a single-stage architecture using a frozen convolutional backbone.

The Problem: Two-Stage Redundancy and ViT Scaling

Existing state-of-the-art OVS methods suffer from two primary drawbacks:

  1. Computational Inefficiency: Extracting features from the same image multiple times (once for mask generation and once for classification) is redundant. Prior models like ODISE are powerful but slow, making them difficult to deploy in real-time scenarios.
  2. Resolution Sensitivity in ViTs: Most CLIP models utilize Vision Transformer (ViT) backbones. While effective at their native pre-training resolution (usually 224px), ViT features often become noisy or "pixelated" when scaled up to the high resolutions (1024px+) required for accurate dense segmentation. This degradation limits the quality of the generated masks.

Methods: The FC-CLIP Framework

FC-CLIP proposes a streamlined approach by utilizing a single, frozen convolutional CLIP backbone (specifically ConvNeXt-L). The framework is built on three core pillars:

1. Shared Frozen Convolutional Backbone

Instead of fine-tuning the backbone, which can destroy the delicate image-text alignment learned during CLIP pre-training, FC-CLIP keeps the backbone frozen. The authors demonstrate that convolutional backbones possess an inductive bias that allows them to generalize to high resolutions much more gracefully than ViTs. The resulting features remain semantically coherent and "smooth," making them ideal for both mask generation and classification.

2. Dual Classifier System

To balance specialized knowledge with general open-vocabulary capabilities, FC-CLIP employs two classifiers:

  • In-Vocabulary Classifier: A lightweight module trained on the base dataset (e.g., COCO) to accurately identify "seen" classes.
  • Out-of-Vocabulary Classifier: This module performs mask-pooling directly over the frozen CLIP features. Because the features are preserved in their original CLIP space, they retain the ability to recognize "unseen" classes through text-image similarity.

3. Geometric Ensemble

During the inference stage, the model uses a geometric ensemble to combine the scores from both classifiers. This ensures that the model can leverage the high precision of the trained in-vocabulary classifier while still benefiting from the broad knowledge base of the CLIP-based out-of-vocabulary classifier.

Experimental Results

FC-CLIP was trained on COCO panoptic data and evaluated in a zero-shot manner across several diverse benchmarks, including ADE20K, Mapillary Vistas, and Cityscapes.

Performance and Efficiency

The results demonstrate that FC-CLIP is not only more accurate but also significantly more efficient than previous two-stage models:

  • Accuracy: On ADE20K, FC-CLIP achieved 26.8 PQ, outperforming the previous state-of-the-art (ODISE) by +4.2 PQ. It also showed remarkable gains on Cityscapes (+20.1 PQ).
  • Speed: FC-CLIP is 6.6x faster during inference and 7.5x faster during training compared to ODISE.
  • Model Size: The architecture is highly compact, utilizing 5.9x fewer parameters than the prior art.

Discussion and Insights

The success of FC-CLIP suggests that "convolutions die hard" in the context of dense prediction. While Transformers have dominated many vision tasks, the spatial inductive biases of convolutional architectures provide a distinct advantage when scaling to the high resolutions necessary for segmentation. Furthermore, the study proves that a well-designed single-stage framework can outperform more complex two-stage systems by maintaining the integrity of frozen CLIP features.

Limitations

Despite its strengths, FC-CLIP has areas for further refinement. The "Segmentation Quality" (SQ) metric is slightly lower than some heavier models, indicating that while the framework is efficient, there is still room to improve the fine-grained details of the mask generation. Additionally, like all CLIP-based models, FC-CLIP may inherit biases from the large-scale internet data used during pre-training and can occasionally struggle with hierarchical category conflicts (e.g., distinguishing a "part" from a "whole").

Conclusion

FC-CLIP represents a significant shift in open-vocabulary segmentation research. By moving to a single-stage, frozen convolutional architecture, it achieves a new state-of-the-art in performance while drastically reducing the computational footprint. This work paves the way for more practical and efficient universal vision systems.

Summary updated

Abstract

Open-vocabulary segmentation is a challenging task requiring segmenting and recognizing objects from an open set of categories. One way to address this challenge is to leverage multi-modal models, such as CLIP, to provide image and text features in a shared embedding space, which bridges the gap between closed-vocabulary and open-vocabulary recognition. Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through a mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using a shared Frozen Convolutional CLIP backbone, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields a better accuracy-cost trade-off. The proposed FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining. When training on COCO panoptic data only and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Additionally, the training and testing time of FC-CLIP is 7.5x and 6.6x significantly faster than the same prior art, while using 5.9x fewer parameters. FC-CLIP also sets a new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code at https://github.com/bytedance/fc-clip
BibTeX
@inproceedings{yu2023convolutions,
  author = {Qihang Yu and Ju He and Xueqing Deng and Xiaohui Shen and Liang-Chieh Chen},
  title = {{Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=83LJRUzXWj},
}
Figure