Abstract
Open-vocabulary segmentation aims to identify and delineate objects from arbitrary categories, even those not seen during training. While Vision-Language Models (VLMs) like CLIP provide a strong foundation, adapting them for pixel-level tasks often leads to overfitting on training styles or loss of generalization. This paper introduces the Generalization Boosted Adapter (GBA), a plug-and-play module that enhances CLIP-based segmenters. GBA utilizes a Style Diversification Adapter (SDA) to decouple style from content via Fourier transforms and a Correlation Constraint Adapter (CCA) to refine text-image associations in the frequency domain. Experimental results demonstrate state-of-the-art performance on benchmarks like COCO and ADE20K with minimal computational overhead.
Introduction
The advent of Vision-Language Models (VLMs) has significantly advanced the field of open-vocabulary recognition. However, translating image-level understanding to pixel-level segmentation remains challenging. Two primary obstacles exist: the granularity gap, where CLIP lacks the fine-grained spatial awareness for precise boundaries, and the overfitting trap, where fine-tuning on specific datasets causes the model to memorize textures and backgrounds, compromising its ability to generalize to novel categories.
Methodology
The Generalization Boosted Adapter (GBA) addresses these challenges through two synergistic components integrated into a standard segmentation pipeline (such as FC-CLIP).
Style Diversification Adapter (SDA)
The SDA is designed for the early layers of the network to combat overfitting to training data styles. It employs a Fourier Transform to separate image features into amplitude (representing style, such as lighting and texture) and phase (representing structural content). By applying subtle jitter to the amplitude while preserving the phase, the SDA generates synthetic style variations. This process encourages the model to focus on invariant structural features rather than specific environmental conditions.
Correlation Constraint Adapter (CCA)
Implemented in deeper layers, the CCA refines the mapping between text categories and image regions using a cross-attention mechanism. To prevent false associations with background clutter, the CCA performs normalization in the frequency domain. This suppresses low-frequency noise (background) and amplifies high-frequency details (object edges), ensuring more precise semantic alignment.
Experimental Results
GBA was evaluated on diverse benchmarks, including COCO, ADE20K, and the MESS cross-domain suite.
- COCO Panoptic: GBA achieved a Panoptic Quality (PQ) of 57.5%, a 3.1% improvement over the FC-CLIP baseline.
- ADE20K-847: On this challenging long-tail dataset, it reached a record 15.1% mIoU.
- Efficiency: Notably, these gains were achieved with only a 3.23% increase in parameters, maintaining inference speeds (FPS) comparable to the baseline.
Conclusion and Future Work
The GBA provides a robust and efficient solution for open-vocabulary segmentation by mathematically separating style from content and filtering semantic noise. While highly effective in natural scenes, future work will address challenges in extreme domain gaps (e.g., aerial imagery) and hierarchical label disambiguation (e.g., distinguishing a "dog" from a "dog tail").
Summary updated