Abstract
Open-vocabulary semantic segmentation (OVS) aims to identify and delineate objects based on arbitrary text descriptions. While Vision-Language Models (VLMs) like CLIP provide a strong foundation, they suffer from a "granularity gap" between image-level pretraining and pixel-level segmentation requirements. FGAseg bridges this gap through a novel framework featuring a bidirectional pixel-text alignment transformer and a category supplementation module that treats text tokens as dynamic convolutional kernels. FGAseg achieves state-of-the-art performance across multiple benchmarks, including ADE20K and PASCAL-Context, demonstrating superior boundary definition and zero-shot generalization.
Introduction
Traditional semantic segmentation models are limited to a fixed set of categories. Open-vocabulary segmentation (OVS) breaks these boundaries by leveraging the joint embedding space of VLMs. However, most existing methods struggle with the fundamental mismatch in spatial resolution: VLMs are optimized to match whole images to captions, whereas segmentation requires precise local alignment. This often results in coarse masks that fail to capture fine-grained object boundaries.
FGAseg addresses this by introducing a single-stage framework that refines coarse VLM features into high-resolution, category-aware representations through explicit pixel-level alignment and local category supplementation.
Methodology
The FGAseg architecture consists of three core components: a Pixel-Level Alignment Module, a Category Supplementation Propagation Module, and a Category-aware Decoder.
1. Pixel-Level Alignment Module
To refine the initial features from the image encoder, FGAseg employs the P2Tformer (Pixel-Text Alignment Transformer). This module uses cross-modal attention to transfer visual information into the text-based semantic space at a granular level. To ensure this alignment remains faithful to the original VLM's knowledge, the T2Ploss (Text-Pixel Alignment Loss) is introduced, enforcing bidirectional correspondence between specific image regions and their corresponding text descriptions.
2. Category Supplementation Propagation Module
This module enhances the feature map with category-specific context using two complementary strategies:
- Global Category Supplementation (GCS): Calculates standard cosine similarity to capture broad semantic context across the image.
- Local Category Supplementation (LCS): In a novel approach, text tokens are transformed into convolutional kernels. These kernels slide over the image features to calculate local similarity, effectively acting as high-resolution pseudo-masks that provide sharp boundary cues.
3. Category-aware Decoder
The decoder integrates the similarity matrices (pseudo-masks) into a hierarchical upsampling process. To maintain efficiency during inference—especially when dealing with thousands of potential classes—the decoder utilizes Top-K selection. This ensures the model only processes the most relevant categories for any given pixel, significantly reducing computational overhead without sacrificing accuracy.
Experimental Results
FGAseg was trained on the COCO-Stuff dataset and evaluated across five major benchmarks to test its zero-shot generalization capabilities.
Quantitative Performance
The model consistently outperformed existing state-of-the-art methods such as CAT-Seg and SAN:
- ADE20K (A-847): Achieved 16.3 mIoU using a ViT-L/14 backbone, setting a new benchmark for large-scale open-vocabulary tasks.
- PASCAL-Context (PC-459): Reached 23.9 mIoU, demonstrating robustness in high-cardinality label spaces.
- PASCAL VOC (PAS-20): Achieved an impressive 97.1 mIoU, showing that the model excels at standard categories while maintaining open-vocabulary flexibility.
Qualitative Analysis
Visualizations confirm that the LCS module successfully generates sharp pseudo-masks. The model effectively delineates complex boundaries for both small objects (e.g., "mouse") and large background regions (e.g., "sky"), overcoming the typical "blurriness" associated with CLIP-based segmentation.
Conclusion
FGAseg demonstrates that fine-grained alignment is the key to unlocking the full potential of VLMs for semantic segmentation. By treating text as a spatial filter through local convolutional kernels, the framework provides a scalable and accurate solution for open-vocabulary tasks. Future work will explore extending this bidirectional alignment to multimodal inputs beyond text, such as audio-guided segmentation.
Summary updated