Abstract
Open-vocabulary segmentation (OVS) is a challenging task requiring the segmentation and recognition of objects from an open set of categories. To bridge the gap between closed-vocabulary and open-vocabulary recognition, existing methods typically leverage multi-modal models like CLIP. However, most current approaches adopt a two-stage framework: a mask generator first proposes regions, which are then fed into a CLIP model for classification. This process is computationally inefficient, requiring multiple feature extraction passes. In this paper, we propose FC-CLIP, a single-stage framework built on a shared Frozen Convolutional CLIP backbone. FC-CLIP simplifies the pipeline while achieving a superior accuracy-cost trade-off. Our observations indicate that a frozen convolutional CLIP backbone not only maintains open-vocabulary classification capabilities but also serves as a robust mask generator that generalizes better to high resolutions than Vision Transformer (ViT) alternatives. When trained only on COCO panoptic data, FC-CLIP achieves state-of-the-art zero-shot performance across ADE20K, Mapillary Vistas, and Cityscapes, while being significantly faster and more parameter-efficient than prior art.
Introduction
Open-vocabulary segmentation (OVS) represents a significant leap toward universal scene understanding, aiming to recognize and segment any object, including those not encountered during training. The primary catalyst for progress in this field has been Contrastive Language-Image Pre-training (CLIP), which aligns image and text features in a shared embedding space.
Despite the power of CLIP, integrating it into dense prediction tasks like segmentation has historically led to complex, two-stage architectures. These systems typically use a dedicated mask generator (e.g., Mask2Former) to identify regions and then crop those regions to be processed by a separate CLIP encoder. This redundancy results in high latency and massive parameter counts. FC-CLIP addresses these inefficiencies by unifying the pipeline into a single-stage architecture using a frozen convolutional backbone.
The Problem: Two-Stage Redundancy and ViT Scaling
Existing state-of-the-art OVS methods suffer from two primary drawbacks:
- Computational Inefficiency: Extracting features from the same image multiple times (once for mask generation and once for classification) is redundant. Prior models like ODISE are powerful but slow, making them difficult to deploy in real-time scenarios.
- Resolution Sensitivity in ViTs: Most CLIP models utilize Vision Transformer (ViT) backbones. While effective at their native pre-training resolution (usually 224px), ViT features often become noisy or "pixelated" when scaled up to the high resolutions (1024px+) required for accurate dense segmentation. This degradation limits the quality of the generated masks.
Methods: The FC-CLIP Framework
FC-CLIP proposes a streamlined approach by utilizing a single, frozen convolutional CLIP backbone (specifically ConvNeXt-L). The framework is built on three core pillars:
1. Shared Frozen Convolutional Backbone
Instead of fine-tuning the backbone, which can destroy the delicate image-text alignment learned during CLIP pre-training, FC-CLIP keeps the backbone frozen. The authors demonstrate that convolutional backbones possess an inductive bias that allows them to generalize to high resolutions much more gracefully than ViTs. The resulting features remain semantically coherent and "smooth," making them ideal for both mask generation and classification.
2. Dual Classifier System
To balance specialized knowledge with general open-vocabulary capabilities, FC-CLIP employs two classifiers:
- In-Vocabulary Classifier: A lightweight module trained on the base dataset (e.g., COCO) to accurately identify "seen" classes.
- Out-of-Vocabulary Classifier: This module performs mask-pooling directly over the frozen CLIP features. Because the features are preserved in their original CLIP space, they retain the ability to recognize "unseen" classes through text-image similarity.
3. Geometric Ensemble
During the inference stage, the model uses a geometric ensemble to combine the scores from both classifiers. This ensures that the model can leverage the high precision of the trained in-vocabulary classifier while still benefiting from the broad knowledge base of the CLIP-based out-of-vocabulary classifier.
Experimental Results
FC-CLIP was trained on COCO panoptic data and evaluated in a zero-shot manner across several diverse benchmarks, including ADE20K, Mapillary Vistas, and Cityscapes.
Performance and Efficiency
The results demonstrate that FC-CLIP is not only more accurate but also significantly more efficient than previous two-stage models:
- Accuracy: On ADE20K, FC-CLIP achieved 26.8 PQ, outperforming the previous state-of-the-art (ODISE) by +4.2 PQ. It also showed remarkable gains on Cityscapes (+20.1 PQ).
- Speed: FC-CLIP is 6.6x faster during inference and 7.5x faster during training compared to ODISE.
- Model Size: The architecture is highly compact, utilizing 5.9x fewer parameters than the prior art.
Discussion and Insights
The success of FC-CLIP suggests that "convolutions die hard" in the context of dense prediction. While Transformers have dominated many vision tasks, the spatial inductive biases of convolutional architectures provide a distinct advantage when scaling to the high resolutions necessary for segmentation. Furthermore, the study proves that a well-designed single-stage framework can outperform more complex two-stage systems by maintaining the integrity of frozen CLIP features.
Limitations
Despite its strengths, FC-CLIP has areas for further refinement. The "Segmentation Quality" (SQ) metric is slightly lower than some heavier models, indicating that while the framework is efficient, there is still room to improve the fine-grained details of the mask generation. Additionally, like all CLIP-based models, FC-CLIP may inherit biases from the large-scale internet data used during pre-training and can occasionally struggle with hierarchical category conflicts (e.g., distinguishing a "part" from a "whole").
Conclusion
FC-CLIP represents a significant shift in open-vocabulary segmentation research. By moving to a single-stage, frozen convolutional architecture, it achieves a new state-of-the-art in performance while drastically reducing the computational footprint. This work paves the way for more practical and efficient universal vision systems.
Summary updated