Abstract
Open-Vocabulary Segmentation (OVS) enables AI to segment objects based on arbitrary text descriptions, but the reliance on massive pre-trained backbones like CLIP makes these models resource-intensive. This paper introduces OpenTrans, a framework that achieves high-performance OVS with significantly reduced model size and training costs. OpenTrans identifies "transferable" sparse subnetworks through semantic-agnostic pruning and utilizes a principled "heavy-tail" analysis to selectively fine-tune only the most critical layers. Experimental results demonstrate that OpenTrans can reduce parameters by nearly 50% and training FLOPs to 32.6% of the baseline while maintaining or improving mIoU across multiple benchmarks.
Introduction
Open-Vocabulary Segmentation (OVS) is a powerful capability, allowing models to segment objects even if they weren't explicitly seen during training, provided a text description is available. Most state-of-the-art OVS models inherit large pre-trained backbones (e.g., CLIP) to leverage their broad visual-semantic knowledge. However, this approach introduces two significant challenges: massive model size, which hinders deployment on edge devices, and expensive fine-tuning requirements. Existing efficiency techniques like pruning are often heuristic-based and framework-specific. OpenTrans addresses these issues by proposing a "one design, transferable to all" efficiency strategy.
The OpenTrans Framework
OpenTrans employs a two-step approach to optimize OVS models without sacrificing their open-vocabulary nature.
Semantic-Agnostic Model Pruning
To find a smaller subnetwork that remains effective across different OVS frameworks (e.g., DeepLabV3, Mask2Former), the authors use Iterative Magnitude Pruning (IMP). Uniquely, this pruning is guided solely by a knowledge distillation loss from CLIP, avoiding any class-specific semantic supervision. This ensures the resulting sparse backbone retains generalizable features rather than overfitting to a specific set of training classes. This approach allows the same sparse backbone to be "transferred" across different segmentation architectures without re-running the expensive pruning process.
Principled Fine-tuning via Heavy-tail Analysis
Instead of the common heuristic of freezing early layers, OpenTrans uses Empirical Spectral Density (ESD) to analyze the weight matrices of each layer. By calculating a power-law coefficient ($\alpha$) for each layer, the model can distinguish between "well-trained" layers (small $\alpha$) and "under-trained" layers (large $\alpha$). Only the under-trained layers are updated during fine-tuning, which saves computation and prevents the "catastrophic forgetting" of CLIP's general knowledge. This provides a mathematical, data-driven basis for layer selection rather than relying on trial and error.
Experimental Evaluation
The models were trained on COCO-Panoptic and evaluated on several zero-shot benchmarks to test transferability, including ADE20K (150 and 847 classes), Cityscapes, Pascal VOC 2012, and Pascal Context (PC-59 and PC-459).
Key Results
OpenTrans demonstrates significant gains in efficiency and performance. On the Han et al. framework, parameters were reduced from 44.1M to 22.9M (~48% reduction) and FLOPs from 268.2G to 173.3G with minimal accuracy loss.
Furthermore, combining pruning with selective fine-tuning reduced training FLOPs to just 32.6% of the original baseline. In some instances, such as with the FC-CLIP framework on ADE20K-150, the compressed model actually saw a 2% mIoU boost, suggesting that principled freezing helps regularize the model and preserve pre-trained knowledge.
Conclusion
OpenTrans proves that OVS efficiency can be achieved through principled, transferable methods rather than framework-specific heuristics. By identifying framework-agnostic sparse subnetworks and using heavy-tail analysis for fine-tuning, the framework significantly lowers the barrier for deploying advanced segmentation models. While the current work focuses on ResNet-50 backbones, future research could extend these techniques to larger convolutional architectures, Vision Transformers, and other open-vocabulary tasks like object detection.
Summary updated