Generalization Boosted Adapter for Open-Vocabulary Segmentation

Wenhao Xu, Changwei Wang, Xuxiang Feng, Rongtao Xu, Longzhao Huang, Zherui Zhang, Li Guo, Shibiao Xu

TL;DR

The Generalization Boosted Adapter (GBA) is a novel "plug-and-play" module designed to help Vision-Language Models (VLMs) like CLIP excel at segmenting objects they’ve never seen before. By using Fourier transforms to diversify image styles and cross-attention to filter out background noise, GBA significantly improves a model's ability to generalize to new categories without the typical "overfitting" that plagues fine-tuned models.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like CLIP have revolutionized computer vision by learning to associate images with text across massive datasets. However, turning these "image-level" recognizers into "pixel-level" segmenters is notoriously difficult for two reasons:

  1. The Granularity Gap: CLIP is trained to understand whole images, not individual pixels. It lacks the fine-grained detail needed to draw precise boundaries around objects.
  2. The Overfitting Trap: When researchers try to "fine-tune" CLIP for segmentation using smaller, specialized datasets, the model often "memorizes" the specific textures and backgrounds of that data. This destroys its "open-vocabulary" magic—the ability to recognize objects outside of its training set.

The authors of this paper aim to bridge this gap, creating a model that is both precise and robust enough to handle the "infinite" variety of objects found in the real world.

Methods

The authors introduce the Generalization Boosted Adapter (GBA), which consists of two specialized components integrated into a standard CLIP-based segmentation pipeline (specifically using FC-CLIP as a baseline).

  • Style Diversification Adapter (SDA): This module operates in the early layers of the network. It uses a Fourier Transform to decouple image features into two parts: Amplitude (which represents "style" like lighting and texture) and Phase (which represents "content" or the actual shapes). By subtly jittering the amplitude while keeping the phase intact, the SDA creates "synthetic" style variations. This forces the model to focus on the object's structure rather than memorizing a specific lighting condition or texture.
  • Correlation Constraint Adapter (CCA): Placed in the deeper layers, the CCA uses a cross-attention mechanism to link text categories directly to specific image regions. Crucially, it performs a normalization step in the frequency domain to suppress low-frequency "noise" (like blurry backgrounds) and amplify high-frequency details (like sharp object edges). This ensures the model doesn't make "false associations" between a label and the background.

Both modules are lightweight and designed to be "plug-and-play," meaning they can be added to existing architectures with minimal computational cost.

Datasets

The model was trained and evaluated on a wide array of standard and challenging benchmarks:

  • COCO & ADE20K: Used for both panoptic and semantic segmentation. ADE20K-847 is particularly notable for having a massive "long-tail" of 847 different object categories.
  • PASCAL Context & PASCAL VOC: Classic benchmarks for testing semantic segmentation accuracy.
  • MESS Benchmark (BDD-100K, UAvid, ISPRS Potsdam): These were used to test "cross-domain" generalization, involving driving scenes, aerial imagery, and remote sensing.

Key Insights and Ideas

  • Style is the Enemy of Generalization: The authors realized that models often overfit to the style of the training data. By mathematically separating style from content using Fourier transforms, they can augment the model's "experience" without needing more labeled data.
  • Frequency Matters: Not all pixels are created equal. Low-frequency components often represent background "clutter," while high-frequency components contain the semantic "meat" of an object (its edges and contours). Filtering for high frequencies during the text-image matching phase leads to much cleaner segmentation.
  • Shallow vs. Deep Synergy: The SDA works best in shallow layers (where style is processed), while the CCA works best in deep layers (where semantic meaning is finalized). Together, they create a "synergistic effect" that neither could achieve alone.

Key Results

The GBA method achieved state-of-the-art (SOTA) results across multiple benchmarks:

  • COCO Panoptic: Achieved a Panoptic Quality (PQ) of 57.5%, outperforming the previous best (FC-CLIP) by 3.1%.
  • ADE20K-847: On this extremely difficult "long-tail" dataset, GBA reached 15.1% mIoU, setting a new record for open-vocabulary performance.
  • Efficiency: Despite the performance boost, GBA only increases total parameters by 3.23% and maintains an inference speed (FPS) comparable to the baseline, making it practical for real-world use.

Limitations and Future Work

While GBA is a significant step forward, the authors note a few remaining hurdles:

  • Occlusion and Camouflage: The model still struggles when objects are heavily hidden or blend perfectly into their surroundings.
  • Extreme Domain Gaps: While it handles natural scenes well, the performance drop on aerial and remote sensing imagery (UAvid/Potsdam) suggests that the "viewpoint" and "scale" differences in these domains still present a challenge for CLIP-based models.
  • Vocabulary Overlap: Future work will look into distinguishing between hierarchically related labels (e.g., knowing the difference between a "dog" and a "dog tail" when both are present).

Summary updated

Abstract

Vision-language models (VLMs) have demonstrated remarkable open-vocabulary object recognition capabilities, motivating their adaptation for dense prediction tasks like segmentation. However, directly applying VLMs to such tasks remains challenging due to their lack of pixel-level granularity and the limited data available for fine-tuning, leading to overfitting and poor generalization. To address these limitations, we propose Generalization Boosted Adapter (GBA), a novel adapter strategy that enhances the generalization and robustness of VLMs for open-vocabulary segmentation. GBA comprises two core components: (1) a Style Diversification Adapter (SDA) that decouples features into amplitude and phase components, operating solely on the amplitude to enrich the feature space representation while preserving semantic consistency; and (2) a Correlation Constraint Adapter (CCA) that employs cross-attention to establish tighter semantic associations between text categories and target regions, suppressing irrelevant low-frequency “noise” information and avoiding erroneous associations. Through the synergistic effect of the shallow SDA and the deep CCA, GBA effectively alleviates overfitting issues and enhances the semantic relevance of feature representations. As a simple, efficient, and plug-and-play component, GBA can be flexibly integrated into various CLIP-based methods, demonstrating broad applicability and achieving state-of-the-art performance on multiple open-vocabulary segmentation benchmarks. Code are available at https://github.com/clearxu/BGA.
BibTeX
@article{xu2025generalization,
  author = {Wenhao Xu and Changwei Wang and Xuxiang Feng and Rongtao Xu and Longzhao Huang and Zherui Zhang and Li Guo and Shibiao Xu},
  title = {{Generalization Boosted Adapter for Open-Vocabulary Segmentation}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  url = {https://openreview.net/forum?id=tmNEWyLYsA},
}
Figure