Transferable and Principled Efficiency for Open-Vocabulary Segmentation

Jingxuan Xu, Wuyang Chen, Yao Zhao, Yunchao Wei

TL;DR

Open-Vocabulary Segmentation (OVS) allows models to segment arbitrary objects using natural language descriptions, but relying on massive foundation models like CLIP makes it computationally expensive to train and deploy. This paper introduces OpenTrans, a framework that prunes heavy vision-language backbones in a semantic-agnostic way so the resulting lightweight subnetwork can be seamlessly transferred across different OVS architectures without re-training. By combining this with a principled layer-freezing strategy based on weight spectrum analysis, OpenTrans slashes both model size and training costs while maintaining or even improving segmentation accuracy.

What Problem Does This Paper Address?

Traditional semantic segmentation models are limited to a fixed set of predefined categories seen during training. Open-Vocabulary Segmentation (OVS) breaks this barrier by leveraging vision-language models (like CLIP) to recognize novel categories from arbitrary text prompts.

However, OVS comes with a massive computational tax:

  1. Huge Model Sizes: The heavy image encoders from foundation models must be kept intact to preserve their open-world generalization capabilities.
  2. Expensive Fine-Tuning: Adapting these models to segmentation tasks requires heavy computational overhead.

While standard model compression (pruning) and parameter-efficient fine-tuning (like adapters) can help, they are usually highly heuristic and framework-specific. If you prune a backbone for one segmentation architecture, you cannot easily transfer that sparse backbone to another without re-running the expensive pruning process from scratch. The authors of this paper ask a fundamental question: Can we design principled efficiency methods that are seamlessly transferable across different OVS frameworks?

Methods

OpenTrans achieves "one design, transferable to all" efficiency through a two-step approach:

Figure 2. Overview of OpenTrans(Ours). We introduce principled and transferable efficiency in two folds. Step1: we prune the heavy CLIP image encoder without semantic awareness, this turns the backbone into a semantic-agnostic transferable subnetwork. This allows us to seamlessly transfer subnetwork to other OVS Frameworks, such as Deeplabv3 [2], Mask2former [6] and FC-CLIP [47]. Step2: during fine-tuning, we further explore and prioritize principled efficiency by introducing layer-wise heavy-tail spectrum analysis. This method involves selectively updating layers with light-tail spectra in their pretrained weights, while keeping layers with heavy-tail spectra frozen.

1. Semantic-Agnostic Model Pruning (Model Efficiency)

Instead of pruning the backbone using standard segmentation datasets (which causes the model to overfit to seen classes and creates framework-specific masks), OpenTrans prunes the CLIP image encoder using only a knowledge distillation loss. This loss aligns the text and vision spaces without relying on semantic class labels. Because the resulting sparse subnetwork is entirely semantic-agnostic, it can be directly plugged into completely different OVS pipelines (such as CNN-based, Mask2Former-based, or multi-classifier frameworks) without any modifications or extra pruning costs.

Figure 3. We transfer our semantic-agnostic sparse masks to different OVS pipelines. Left: Traditional CNN-based frame- works: these frameworks commonly employ simple upsampling techniques. Middle: Mask2Former-based frameworks, the ap- proach utilizes a decoupled mask and class segmentation head. Right: OVS with extra classifiers: some frameworks propose to leverage a frozen CLIP classifier to further utilize the pretrained visual knowledge (with the blue box indicating frozen classifier layers of the model).

2. Layer Selection via Heavy-Tail Analysis (Training Efficiency)

To optimize the fine-tuning phase, the authors avoid the brute-force approach of updating every single layer. Instead, they analyze the Empirical Spectral Density (ESD) of the pretrained weight matrices.

According to heavy-tail theory, well-trained layers that have effectively captured data correlations exhibit a "heavy-tail" distribution in their eigenvalues (characterized by a smaller power-law coefficient $\alpha$). OpenTrans calculates these $\alpha$ values, freezes the top 50% highest-quality (heavy-tail) layers, and only calculates gradients for the remaining under-trained (light-tail) layers. This data-free, zero-overhead analysis tells the framework exactly where to allocate training compute.

Datasets

The subnetwork is initially trained on COCO-Panoptic (133 classes) using the semantic-agnostic distillation loss. To evaluate its cross-dataset, zero-shot generalization capabilities, it is tested across several diverse benchmarks:

  • Cityscapes: For urban scene understanding.
  • ADE20K: Evaluated on both the standard ADE20K-150 split and the highly challenging ADE20K-847 split.
  • PASCAL VOC 2012 (PAS-20)
  • Pascal Context: Evaluated on both the PC-59 and PC-459 splits.

Key Insights and Ideas

  • Pruning via Distillation Preserves the Open World: Standard pruning often strips away a foundation model's zero-shot capabilities. By framing pruning purely as a text-to-image space distillation task, OpenTrans shrinks the model while keeping its open-vocabulary magic alive.
  • The Weight Spectrum Predicts Trainability: You don't need to run forward or backward passes over data to know which layers are already "smart." The eigenvalue distribution of a layer's weight matrix serves as a reliable, static proxy for its optimization quality.
  • Freezing "Good" Layers Prevents Overfitting: Counterintuitively, freezing the layers with small $\alpha$ values actually improves zero-shot performance on novel classes. It acts as a regularizer, preventing the model from destroying its pre-trained general knowledge during downstream fine-tuning.

Key Results

OpenTrans demonstrates an exceptional trade-off between segmentation accuracy (mIoU) and computational cost across multiple popular OVS baselines (Han et al., DeepLabV3, and FC-CLIP).

Performance vs. Efficiency trade-off
  • Subnetwork Transferability: When transferring the exact same pruned subnetwork to the Han et al. baseline, parameters dropped by 48.1% (from 44.1M to 22.9M) and inference FLOPs dropped by 35.4% (from 268.2G to 173.3G), with only a marginal drop in accuracy.
  • Surpassing Baselines: When applied to DeepLabV3, OpenTrans successfully halved the model size while actually improving mIoU on Cityscapes by 4% and ADE20K-150 by 2%.
  • Training Cost Reductions: The layer-selective fine-tuning strategy alone reduced training FLOPs by 12.0%. When combined with the pruned subnetwork, total training compute dropped by 32.6% (from 181.4P to 122.2P FLOPs).

Limitations and Future Work

The authors note a few areas for improvement and future exploration:

  1. Backbone Architectures: The current work primarily focuses on ResNet-50 backbones. Future work will explore adapting this method to larger convolutional networks (like ConvNeXt-Large) and Vision Transformers (ViTs).
  2. Granularity: The layer-wise selection approach is relatively coarse. A more fine-grained selection method operating at the weight element or channel level could yield even higher efficiency.
  3. Task Expansion: The core principles of transferable, semantic-agnostic pruning could be extended beyond segmentation to other open-vocabulary tasks, such as open-set object detection.

Summary updated

Abstract

Recent success of pre-trained foundation vision-language models makes Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance, this approach introduces heavy computational overheads for two challenges: 1) large model sizes of the backbone; 2) expensive costs during the fine-tuning. These challenges hinder this OVS strategy from being widely applicable and affordable in real-world scenarios. Although traditional methods such as model compression and efficient fine-tuning can address these challenges, they often rely on heuristics. This means that their solutions cannot be easily transferred and necessitate re-training on different models, which comes at a cost. In the context of efficient OVS, we target achieving performance that is comparable to or even better than prior OVS works based on large vision-language foundation models, by utilizing smaller models that incur lower training costs. The core strategy is to make our efficiency principled and thus seamlessly transferable from one OVS framework to others without further customization. Comprehensive experiments on diverse OVS benchmarks demonstrate our superior trade-off between segmentation accuracy and computation costs over previous works. Our code is available on https://github.com/Xujxyang/OpenTrans
BibTeX
@misc{xu2024transferable,
  author = {Jingxuan Xu and Wuyang Chen and Yao Zhao and Yunchao Wei},
  title = {{Transferable and Principled Efficiency for Open-Vocabulary Segmentation}},
  year = {2024},
  eprint = {2404.07448},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.07448},
}
Figure