UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding

Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, Xiaopeng Zhang

TL;DR

UMG-CLIP is a vision-language foundation model that bridges the gap between global image understanding and fine-grained local perception. By training on a massive new dataset (UMG-41M) featuring 389 million region-level pseudo-annotations, it creates a "vision generalist" that achieves state-of-the-art results across tasks ranging from zero-shot classification to dense panoptic segmentation.

What Problem Does This Paper Address?

While models like CLIP have revolutionized vision-language tasks, they suffer from a "granularity gap." Standard CLIP is trained to match a whole image to a single caption, making it excellent at global tasks like image retrieval but relatively poor at "dense" tasks that require localizing specific objects or pixels (like semantic segmentation).

The primary hurdle in fixing this is data: high-quality, large-scale datasets with region-level or pixel-level annotations are incredibly expensive to produce manually. Previous attempts to bridge this gap often relied on noisy "implicit mining" from image-text pairs, which limits the model's ability to distinguish between fine-grained local details.

Methods

The authors introduce UMG-CLIP, a Unified Multi-Granularity framework designed to handle everything from "the big picture" to individual pixels. Their approach rests on three pillars:

  1. Automated Annotation Workflow: Instead of manual labeling, they built a pipeline using a "committee" of specialized models (like SAM for masks, ViTDet for bounding boxes, and BLIP2/Shikra for captions) to generate multi-granularity labels for 41 million existing images.

    Fig. 2: The automated annotation workflow, which generates captions/tags at image- level, region-level, and pixel-level.
  2. Multi-Task Pre-training: UMG-CLIP doesn't just do image-text matching. It uses RoIAlign to extract region features and aligns them with region-specific captions and tags. It employs two types of losses: a contrastive loss for captions and a smooth-L1 loss for tags to improve category differentiation.

  3. Efficiency and Adaptation: To handle the high memory cost of fine-grained training, they use a cluster-based token reduction strategy that cuts visual tokens by 75%. For downstream tasks, they use Parameter-Efficient Tuning (PET) with lightweight "Convpass" modules, keeping the massive pre-trained backbone frozen to preserve its general knowledge.
Fig. 4: UMG-CLIP is a multi-granularity multi-task framework that aligns both image- level and region-level visual features with their corresponding tags and captions. This alignment empowers the model with generalist capabilities across multiple granularity, allowing it to efficiently adapt to various downstream tasks through PET.

Datasets

  • UMG-41M: A new, massive dataset created by re-annotating six public datasets (CC3M, CC12M, SBU, VG, YFCC15M, and ImageNet-21k). It contains 41.41M images and 389.13M regions across 11,741 categories.
  • BigDet Panoptic: A specialized dataset with 3.59M images and 48.6M regions used for dense-level downstream adaptation.
  • Evaluation Benchmarks: The model was tested on 21 classification datasets (including ImageNet variants), COCO and Flickr30k for retrieval, and ADE20K and COCO for segmentation.

Key Insights and Ideas

  • Pseudo-labels at Scale: The paper proves that "good enough" automated labels, when generated at a massive scale (389M regions), can effectively teach a model fine-grained localization that manual datasets are too small to provide.
  • Tags as Anchors: While captions provide rich semantic context, the authors found that adding a tag-based loss acts as a powerful regularizer, helping the model better differentiate between similar object categories.
  • The Power of PET: By using Parameter-Efficient Tuning, the authors showed that a single pre-trained "generalist" backbone can be adapted to vastly different tasks (like classification vs. segmentation) without the need for resource-intensive full fine-tuning.
  • Granularity Matters: Ablation studies showed that adding region-level supervision significantly boosts segmentation performance (+2.2 mIoU) with negligible impact on global classification accuracy.

Key Results

UMG-CLIP demonstrates "generalist" dominance across the board:

  • Zero-shot Classification: UMG-CLIP-E/14 achieved a 74.6% average top-1 accuracy across 21 benchmarks, outperforming the previous SOTA, EVA-02-CLIP, by 1.0%.
  • Retrieval: On COCO, it showed massive gains, improving R@1 by 5.3% for Image-to-Text and 7.0% for Text-to-Image compared to EVA-02-CLIP.
  • Segmentation: It achieved state-of-the-art results in open-vocabulary panoptic segmentation on ADE20K (zero-shot) and COCO.
Fig. 1: Compared to existing models, UMG-CLIP demonstrates outstanding perfor- mance across a wide range of tasks.

Limitations and Future Work

The authors acknowledge that their automated annotation process is "cumbersome," requiring the coordination of multiple heavy-duty models. They suggest that future work should look for better synergy between the data annotation and model training phases to make the pipeline more efficient. Additionally, while the pseudo-labels are effective, they still contain inherent noise that could be further analyzed and mitigated to push performance even higher.

Summary updated

Abstract

Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We believe that UMG-CLIP represents a valuable advancement in vision-language foundation models. The code is available at https://github.com/lygsbw/UMG-CLIP.
BibTeX
@inproceedings{shi2024umg,
  author = {Bowen Shi and Peisen Zhao and Zichen Wang and Yuhang Zhang and Yaoming Wang and Jin Li and Wenrui Dai and Junni Zou and Hongkai Xiong and Qi Tian and Xiaopeng Zhang},
  title = {{UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72920-1\_15},
}
Figure