UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding

Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Li Jin, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, Xiaopeng Zhang

Abstract

Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks.

Introduction: Beyond Global Vision

Most vision-language models, like the original CLIP, are "globalists." They are trained to match an entire image to a single text description. While this works incredibly well for tasks like image classification or search, it leaves the model "blind" to local details. If you ask a standard CLIP model to find a specific small object in a corner or to outline the exact pixels of a chair (segmentation), it often struggles because it never explicitly learned the relationship between specific image regions and specific words.

The challenge in fixing this is data. We have billions of image-caption pairs from the web, but we don't have billions of images where every object is perfectly boxed, masked, and described. Manual labeling at that scale is impossible. UMG-CLIP addresses this by bridging the gap between global image understanding and fine-grained local perception through a unified multi-granularity approach.

Methods: The UMG-CLIP Framework

The authors introduce UMG-CLIP (Unified Multi-Granularity CLIP), which uses several innovations to solve the granularity problem:

Automated Annotation Workflow

Instead of human labels, the researchers used an ensemble of specialized "teacher" models to generate pseudo-annotations. They employed models like RAM for tags, BLIP2 for captions, ViTDet for finding object boxes, and SAM (Segment Anything Model) for pixel-level masks. This allowed them to turn 41 million standard images into a rich, multi-layered dataset.

Multi-Granularity Learning

During training, the model doesn't just look at the whole image. It uses RoIAlign to extract specific regions and aligns those regional features with corresponding text tokens. This dual-path alignment ensures the model understands both the "forest" (the whole image) and the "trees" (individual objects).

The UMG-CLIP learning framework architecture, showcasing image and region alignment losses.

Efficiency and Adaptation

Training on high-resolution images for local tasks is computationally expensive. To save memory, UMG-CLIP uses a cluster-based strategy that identifies and merges redundant visual tokens, reducing the workload by 75% without losing the spatial detail needed for local perception. Furthermore, for downstream tasks like segmentation, the model employs Parameter-Efficient Tuning (PET). By keeping the main model frozen and only training tiny "Convpass" modules, UMG-CLIP preserves its general knowledge while adapting cheaply to new tasks.

Datasets: Scaling Multi-Granularity

The core contribution to the training process is the UMG-41M dataset. It contains 41.41 million images with 389 million regional annotations across 11,741 classes. It was built by re-annotating six existing public datasets (CC3M, CC12M, SBU, VG, YFCC15M, and IN21k). Additionally, a specialized BigDet Panoptic dataset containing 3.59 million images was used for dense adaptation.

Key Results and Insights

UMG-CLIP consistently outperformed standard CLIP and its high-performing variants (like EVA-02-CLIP) across a wide spectrum of tasks.

Performance comparison of UMG-CLIP against state-of-the-art baselines across 21 datasets and multiple task types.
  • Zero-Shot Classification: UMG-CLIP-L/14 achieved an average top-1 accuracy of 72.1% across 21 benchmarks, beating the original OpenAI CLIP by 3.3%.
  • Image Retrieval: On the COCO dataset, it saw a massive +5.3% improvement in Image-to-Text retrieval (R@1) compared to its closest competitors.
  • Segmentation: It achieved state-of-the-art results in open-vocabulary panoptic segmentation, significantly outperforming specialized models like FC-CLIP, even while using fewer parameters.

Insights

  • Pseudo-labels are Powerful: You don't need perfect human labels to reach state-of-the-art performance. If you have enough scale and a smart filtering process to remove low-quality "teacher" guesses, pseudo-annotations are sufficient to teach a model fine-grained localization.
  • Clustering vs. Masking: Many models use random masking to speed up training. The authors found that for local understanding, random masking is destructive. Clustering similar tokens is a much better way to save compute while keeping the "map" of the image intact.
  • The "Generalist" Advantage: By learning image-level, region-level, and pixel-level details simultaneously, the model becomes more robust. Improving its ability to see a "chair" in a small box actually helps it understand a "living room" scene better as a whole.

Conclusion

UMG-CLIP represents a significant advancement in vision-language foundation models by unifying global and local understanding. While the automated annotation process is currently resource-intensive, requiring multiple heavy teacher models, the results prove that multi-granularity alignment is key to creating true vision generalists. Future work will likely focus on making the data generation pipeline more efficient and mitigating noise in pseudo-labels.

The code is available at https://github.com/lygsbw/UMG-CLIP.

Summary updated

Abstract

Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We believe that UMG-CLIP represents a valuable advancement in vision-language foundation models. The code is available at https://github.com/lygsbw/UMG-CLIP.
BibTeX
@inproceedings{shi2024umg,
  author = {Bowen Shi and Peisen Zhao and Zichen Wang and Yuhang Zhang and Yaoming Wang and Li Jin and Wenrui Dai and Junni Zou and Hongkai Xiong and Qi Tian and Xiaopeng Zhang},
  title = {{UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72920-1\_15},
}
Figure