Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Yajie Liu, Pu Ge, Qingjie Liu, Di HuangarXiv 2024 · DOI · arXiv

Abstract

Open-vocabulary semantic segmentation (OVSS) aims to identify and segment arbitrary categories using only text descriptions. Current methods leveraging large-scale image-text pre-training often struggle with the "granularity gap" between coarse training captions and fine-grained inference requirements. This paper introduces Multi-Grained Cross-modal Alignment (MGCA), a framework that bridges this gap by aligning features at object, region, and pixel levels. Furthermore, we propose an adaptive semantic unit to mitigate the common issues of under- and over-segmentation. MGCA achieves new state-of-the-art results on multiple benchmarks, demonstrating superior efficiency and precision using significantly less training data than competing approaches.

Introduction

The "holy grail" of computer vision is the ability to segment any object in an image based on a text description, even for categories unseen during training. While models like CLIP have advanced this field, two major hurdles remain:

  1. The Granularity Gap: Training typically aligns a whole image or region to a caption, whereas inference requires pixel-level precision.
  2. The Semantic Unit Dilemma: Existing methods either group pixels into coarse "blobs" (leading to under-segmentation) or classify pixels independently (leading to noisy over-segmentation).

Methodology

The MGCA framework addresses these issues through two primary innovations: multi-grained alignment during training and an adaptive semantic unit for inference.

Multi-Grained Alignment (Training)

Instead of relying solely on image-level alignment, MGCA explicitly learns at three levels simultaneously:

  • Object-level: Identifies the main object described in the text and aligns its features using a similarity matrix.
  • Region-level: Employs "hard negative mining" to distinguish between matching and non-matching regions, enhancing feature discriminability.
  • Pixel-level: Establishes fine-grained correspondences between specific pixels and words using a "semi-hard positive" strategy to identify the most trustworthy pixels for alignment.

Adaptive Semantic Unit (Inference)

To resolve the under/over-segmentation trade-off, MGCA introduces an adaptive grouping mechanism. By sampling "meta-points" across the image and grouping pixels based on similarity, the model creates "part-level" representations. These units are more stable than individual pixels but more detailed than whole-object groups, maintaining visual coherence without sacrificing precision.

Experimental Results and Insights

MGCA was trained on the CC3M dataset (2.56M pairs) and evaluated on 8 benchmark datasets in a zero-shot setting.

  • State-of-the-Art Performance: MGCA outperformed the previous state-of-the-art (TCL) by 3.5% mIoU on object-oriented datasets and 2.4% mIoU on complex scene datasets like ADE20k and Cityscapes.
  • Efficiency over Scale: The model achieved these results using only ~3 million image-text pairs, outperforming competitors trained on up to 29 million pairs, suggesting that alignment quality is more impactful than raw data volume.
  • Ablation Insights: Pixel-level alignment was found to be the most critical component, contributing a ~15% mIoU improvement over the baseline.
  • Lightweight Architecture: The learnable decoder is highly efficient, with only 4.72M parameters, effectively unlocking the knowledge stored in frozen CLIP encoders.

Conclusion

MGCA demonstrates that effective multi-grained alignment can significantly improve open-vocabulary segmentation performance and efficiency. By bridging the granularity gap and introducing adaptive semantic units, the framework provides a robust solution for learning fine-grained vision tasks from coarse text supervision. While challenges remain regarding inference speed and caption quality, MGCA sets a new benchmark for zero-shot segmentation.

Summary updated

BibTeX
@misc{liu2024multi,
  author = {Yajie Liu and Pu Ge and Qingjie Liu and Di Huang},
  title = {{Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision}},
  year = {2024},
  eprint = {2403.03707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.03707},
}
Figure