FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation

Bingyu Li, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong LiarXiv 2025 · DOI · arXiv

Abstract

Open-vocabulary semantic segmentation (OVS) aims to identify and delineate objects based on arbitrary text descriptions. While Vision-Language Models (VLMs) like CLIP provide a strong foundation, they suffer from a "granularity gap" between image-level pretraining and pixel-level segmentation requirements. FGAseg bridges this gap through a novel framework featuring a bidirectional pixel-text alignment transformer and a category supplementation module that treats text tokens as dynamic convolutional kernels. FGAseg achieves state-of-the-art performance across multiple benchmarks, including ADE20K and PASCAL-Context, demonstrating superior boundary definition and zero-shot generalization.

Introduction

Traditional semantic segmentation models are limited to a fixed set of categories. Open-vocabulary segmentation (OVS) breaks these boundaries by leveraging the joint embedding space of VLMs. However, most existing methods struggle with the fundamental mismatch in spatial resolution: VLMs are optimized to match whole images to captions, whereas segmentation requires precise local alignment. This often results in coarse masks that fail to capture fine-grained object boundaries.

FGAseg addresses this by introducing a single-stage framework that refines coarse VLM features into high-resolution, category-aware representations through explicit pixel-level alignment and local category supplementation.

Methodology

The FGAseg architecture consists of three core components: a Pixel-Level Alignment Module, a Category Supplementation Propagation Module, and a Category-aware Decoder.

Overview of the FGAseg framework, illustrating the flow from image/text encoding through pixel-level alignment and category supplementation to the final decoder.

1. Pixel-Level Alignment Module

To refine the initial features from the image encoder, FGAseg employs the P2Tformer (Pixel-Text Alignment Transformer). This module uses cross-modal attention to transfer visual information into the text-based semantic space at a granular level. To ensure this alignment remains faithful to the original VLM's knowledge, the T2Ploss (Text-Pixel Alignment Loss) is introduced, enforcing bidirectional correspondence between specific image regions and their corresponding text descriptions.

2. Category Supplementation Propagation Module

This module enhances the feature map with category-specific context using two complementary strategies:

  • Global Category Supplementation (GCS): Calculates standard cosine similarity to capture broad semantic context across the image.
  • Local Category Supplementation (LCS): In a novel approach, text tokens are transformed into convolutional kernels. These kernels slide over the image features to calculate local similarity, effectively acting as high-resolution pseudo-masks that provide sharp boundary cues.
Detailed view of Global and Local Category Supplementation. In LCS, text tokens are reshaped into convolutional kernels to extract local similarity maps.

3. Category-aware Decoder

The decoder integrates the similarity matrices (pseudo-masks) into a hierarchical upsampling process. To maintain efficiency during inference—especially when dealing with thousands of potential classes—the decoder utilizes Top-K selection. This ensures the model only processes the most relevant categories for any given pixel, significantly reducing computational overhead without sacrificing accuracy.

Experimental Results

FGAseg was trained on the COCO-Stuff dataset and evaluated across five major benchmarks to test its zero-shot generalization capabilities.

Quantitative Performance

The model consistently outperformed existing state-of-the-art methods such as CAT-Seg and SAN:

  • ADE20K (A-847): Achieved 16.3 mIoU using a ViT-L/14 backbone, setting a new benchmark for large-scale open-vocabulary tasks.
  • PASCAL-Context (PC-459): Reached 23.9 mIoU, demonstrating robustness in high-cardinality label spaces.
  • PASCAL VOC (PAS-20): Achieved an impressive 97.1 mIoU, showing that the model excels at standard categories while maintaining open-vocabulary flexibility.

Qualitative Analysis

Visualizations confirm that the LCS module successfully generates sharp pseudo-masks. The model effectively delineates complex boundaries for both small objects (e.g., "mouse") and large background regions (e.g., "sky"), overcoming the typical "blurriness" associated with CLIP-based segmentation.

Conclusion

FGAseg demonstrates that fine-grained alignment is the key to unlocking the full potential of VLMs for semantic segmentation. By treating text as a spatial filter through local convolutional kernels, the framework provides a scalable and accurate solution for open-vocabulary tasks. Future work will explore extending this bidirectional alignment to multimodal inputs beyond text, such as audio-guided segmentation.

Summary updated

BibTeX
@misc{li2025fgaseg,
  author = {Bingyu Li and Da Zhang and Zhiyuan Zhao and Junyu Gao and Xuelong Li},
  title = {{FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation}},
  year = {2025},
  eprint = {2501.00877},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.00877},
}
Figure