TL;DR
Open-Vocabulary Segmentation (OVS) allows models to segment arbitrary objects using natural language descriptions, but relying on massive foundation models like CLIP makes it computationally expensive to train and deploy. This paper introduces OpenTrans, a framework that prunes heavy vision-language backbones in a semantic-agnostic way so the resulting lightweight subnetwork can be seamlessly transferred across different OVS architectures without re-training. By combining this with a principled layer-freezing strategy based on weight spectrum analysis, OpenTrans slashes both model size and training costs while maintaining or even improving segmentation accuracy.
What Problem Does This Paper Address?
Traditional semantic segmentation models are limited to a fixed set of predefined categories seen during training. Open-Vocabulary Segmentation (OVS) breaks this barrier by leveraging vision-language models (like CLIP) to recognize novel categories from arbitrary text prompts.
However, OVS comes with a massive computational tax:
- Huge Model Sizes: The heavy image encoders from foundation models must be kept intact to preserve their open-world generalization capabilities.
- Expensive Fine-Tuning: Adapting these models to segmentation tasks requires heavy computational overhead.
While standard model compression (pruning) and parameter-efficient fine-tuning (like adapters) can help, they are usually highly heuristic and framework-specific. If you prune a backbone for one segmentation architecture, you cannot easily transfer that sparse backbone to another without re-running the expensive pruning process from scratch. The authors of this paper ask a fundamental question: Can we design principled efficiency methods that are seamlessly transferable across different OVS frameworks?
Methods
OpenTrans achieves "one design, transferable to all" efficiency through a two-step approach:
1. Semantic-Agnostic Model Pruning (Model Efficiency)
Instead of pruning the backbone using standard segmentation datasets (which causes the model to overfit to seen classes and creates framework-specific masks), OpenTrans prunes the CLIP image encoder using only a knowledge distillation loss. This loss aligns the text and vision spaces without relying on semantic class labels. Because the resulting sparse subnetwork is entirely semantic-agnostic, it can be directly plugged into completely different OVS pipelines (such as CNN-based, Mask2Former-based, or multi-classifier frameworks) without any modifications or extra pruning costs.
2. Layer Selection via Heavy-Tail Analysis (Training Efficiency)
To optimize the fine-tuning phase, the authors avoid the brute-force approach of updating every single layer. Instead, they analyze the Empirical Spectral Density (ESD) of the pretrained weight matrices.
According to heavy-tail theory, well-trained layers that have effectively captured data correlations exhibit a "heavy-tail" distribution in their eigenvalues (characterized by a smaller power-law coefficient $\alpha$). OpenTrans calculates these $\alpha$ values, freezes the top 50% highest-quality (heavy-tail) layers, and only calculates gradients for the remaining under-trained (light-tail) layers. This data-free, zero-overhead analysis tells the framework exactly where to allocate training compute.
Datasets
The subnetwork is initially trained on COCO-Panoptic (133 classes) using the semantic-agnostic distillation loss. To evaluate its cross-dataset, zero-shot generalization capabilities, it is tested across several diverse benchmarks:
- Cityscapes: For urban scene understanding.
- ADE20K: Evaluated on both the standard ADE20K-150 split and the highly challenging ADE20K-847 split.
- PASCAL VOC 2012 (PAS-20)
- Pascal Context: Evaluated on both the PC-59 and PC-459 splits.
Key Insights and Ideas
- Pruning via Distillation Preserves the Open World: Standard pruning often strips away a foundation model's zero-shot capabilities. By framing pruning purely as a text-to-image space distillation task, OpenTrans shrinks the model while keeping its open-vocabulary magic alive.
- The Weight Spectrum Predicts Trainability: You don't need to run forward or backward passes over data to know which layers are already "smart." The eigenvalue distribution of a layer's weight matrix serves as a reliable, static proxy for its optimization quality.
- Freezing "Good" Layers Prevents Overfitting: Counterintuitively, freezing the layers with small $\alpha$ values actually improves zero-shot performance on novel classes. It acts as a regularizer, preventing the model from destroying its pre-trained general knowledge during downstream fine-tuning.
Key Results
OpenTrans demonstrates an exceptional trade-off between segmentation accuracy (mIoU) and computational cost across multiple popular OVS baselines (Han et al., DeepLabV3, and FC-CLIP).
- Subnetwork Transferability: When transferring the exact same pruned subnetwork to the Han et al. baseline, parameters dropped by 48.1% (from 44.1M to 22.9M) and inference FLOPs dropped by 35.4% (from 268.2G to 173.3G), with only a marginal drop in accuracy.
- Surpassing Baselines: When applied to DeepLabV3, OpenTrans successfully halved the model size while actually improving mIoU on Cityscapes by 4% and ADE20K-150 by 2%.
- Training Cost Reductions: The layer-selective fine-tuning strategy alone reduced training FLOPs by 12.0%. When combined with the pruned subnetwork, total training compute dropped by 32.6% (from 181.4P to 122.2P FLOPs).
Limitations and Future Work
The authors note a few areas for improvement and future exploration:
- Backbone Architectures: The current work primarily focuses on ResNet-50 backbones. Future work will explore adapting this method to larger convolutional networks (like ConvNeXt-Large) and Vision Transformers (ViTs).
- Granularity: The layer-wise selection approach is relatively coarse. A more fine-grained selection method operating at the weight element or channel level could yield even higher efficiency.
- Task Expansion: The core principles of transferable, semantic-agnostic pruning could be extended beyond segmentation to other open-vocabulary tasks, such as open-set object detection.
Summary updated