Transferable and principled efficiency for open-vocabulary segmentation

Jingxuan Xu, Wuyang Chen, Yao Zhao, Yunchao Wei

Abstract

Open-Vocabulary Segmentation (OVS) enables AI to segment objects based on arbitrary text descriptions, but the reliance on massive pre-trained backbones like CLIP makes these models resource-intensive. This paper introduces OpenTrans, a framework that achieves high-performance OVS with significantly reduced model size and training costs. OpenTrans identifies "transferable" sparse subnetworks through semantic-agnostic pruning and utilizes a principled "heavy-tail" analysis to selectively fine-tune only the most critical layers. Experimental results demonstrate that OpenTrans can reduce parameters by nearly 50% and training FLOPs to 32.6% of the baseline while maintaining or improving mIoU across multiple benchmarks.

Introduction

Open-Vocabulary Segmentation (OVS) is a powerful capability, allowing models to segment objects even if they weren't explicitly seen during training, provided a text description is available. Most state-of-the-art OVS models inherit large pre-trained backbones (e.g., CLIP) to leverage their broad visual-semantic knowledge. However, this approach introduces two significant challenges: massive model size, which hinders deployment on edge devices, and expensive fine-tuning requirements. Existing efficiency techniques like pruning are often heuristic-based and framework-specific. OpenTrans addresses these issues by proposing a "one design, transferable to all" efficiency strategy.

The OpenTrans Framework

OpenTrans employs a two-step approach to optimize OVS models without sacrificing their open-vocabulary nature.

Overview of the OpenTrans framework, illustrating the two-step process of transferable sparse network generation and principled efficient fine-tuning.

Semantic-Agnostic Model Pruning

To find a smaller subnetwork that remains effective across different OVS frameworks (e.g., DeepLabV3, Mask2Former), the authors use Iterative Magnitude Pruning (IMP). Uniquely, this pruning is guided solely by a knowledge distillation loss from CLIP, avoiding any class-specific semantic supervision. This ensures the resulting sparse backbone retains generalizable features rather than overfitting to a specific set of training classes. This approach allows the same sparse backbone to be "transferred" across different segmentation architectures without re-running the expensive pruning process.

Principled Fine-tuning via Heavy-tail Analysis

Instead of the common heuristic of freezing early layers, OpenTrans uses Empirical Spectral Density (ESD) to analyze the weight matrices of each layer. By calculating a power-law coefficient ($\alpha$) for each layer, the model can distinguish between "well-trained" layers (small $\alpha$) and "under-trained" layers (large $\alpha$). Only the under-trained layers are updated during fine-tuning, which saves computation and prevents the "catastrophic forgetting" of CLIP's general knowledge. This provides a mathematical, data-driven basis for layer selection rather than relying on trial and error.

Experimental Evaluation

The models were trained on COCO-Panoptic and evaluated on several zero-shot benchmarks to test transferability, including ADE20K (150 and 847 classes), Cityscapes, Pascal VOC 2012, and Pascal Context (PC-59 and PC-459).

Key Results

OpenTrans demonstrates significant gains in efficiency and performance. On the Han et al. framework, parameters were reduced from 44.1M to 22.9M (~48% reduction) and FLOPs from 268.2G to 173.3G with minimal accuracy loss.

Comparison of OpenTrans (Ours) against popular OVS methods on the Pascal-Context dataset, showing superior mIoU relative to model parameters and FLOPs.

Furthermore, combining pruning with selective fine-tuning reduced training FLOPs to just 32.6% of the original baseline. In some instances, such as with the FC-CLIP framework on ADE20K-150, the compressed model actually saw a 2% mIoU boost, suggesting that principled freezing helps regularize the model and preserve pre-trained knowledge.

Conclusion

OpenTrans proves that OVS efficiency can be achieved through principled, transferable methods rather than framework-specific heuristics. By identifying framework-agnostic sparse subnetworks and using heavy-tail analysis for fine-tuning, the framework significantly lowers the barrier for deploying advanced segmentation models. While the current work focuses on ResNet-50 backbones, future research could extend these techniques to larger convolutional architectures, Vision Transformers, and other open-vocabulary tasks like object detection.

Summary updated

Abstract

Recent success of pre-trained foundation vision-language models makes Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance, this approach introduces heavy computational overheads for two challenges: 1) large model sizes of the backbone; 2) expensive costs during the fine-tuning. These challenges hinder this OVS strategy from being widely applicable and affordable in real-world scenarios. Although traditional methods such as model compression and efficient fine-tuning can address these challenges, they often rely on heuristics. This means that their solutions cannot be easily transferred and necessitate re-training on different models, which comes at a cost. In the context of efficient OVS, we target achieving performance that is comparable to or even better than prior OVS works based on large vision-language foundation models, by utilizing smaller models that incur lower training costs. The core strategy is to make our efficiency principled and thus seamlessly transferable from one OVS framework to others without further customization. Comprehensive experiments on diverse OVS benchmarks demonstrate our superior trade-off between segmentation accuracy and computation costs over previous works
BibTeX
@misc{xu2024transferable,
  author = {Jingxuan Xu and Wuyang Chen and Yao Zhao and Yunchao Wei},
  title = {{Transferable and principled efficiency for open-vocabulary segmentation}},
  year = {2024},
  eprint = {2404.07448},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.07448},
}
Figure