Abstract
Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks.
Introduction: Beyond Global Vision
Most vision-language models, like the original CLIP, are "globalists." They are trained to match an entire image to a single text description. While this works incredibly well for tasks like image classification or search, it leaves the model "blind" to local details. If you ask a standard CLIP model to find a specific small object in a corner or to outline the exact pixels of a chair (segmentation), it often struggles because it never explicitly learned the relationship between specific image regions and specific words.
The challenge in fixing this is data. We have billions of image-caption pairs from the web, but we don't have billions of images where every object is perfectly boxed, masked, and described. Manual labeling at that scale is impossible. UMG-CLIP addresses this by bridging the gap between global image understanding and fine-grained local perception through a unified multi-granularity approach.
Methods: The UMG-CLIP Framework
The authors introduce UMG-CLIP (Unified Multi-Granularity CLIP), which uses several innovations to solve the granularity problem:
Automated Annotation Workflow
Instead of human labels, the researchers used an ensemble of specialized "teacher" models to generate pseudo-annotations. They employed models like RAM for tags, BLIP2 for captions, ViTDet for finding object boxes, and SAM (Segment Anything Model) for pixel-level masks. This allowed them to turn 41 million standard images into a rich, multi-layered dataset.
Multi-Granularity Learning
During training, the model doesn't just look at the whole image. It uses RoIAlign to extract specific regions and aligns those regional features with corresponding text tokens. This dual-path alignment ensures the model understands both the "forest" (the whole image) and the "trees" (individual objects).
Efficiency and Adaptation
Training on high-resolution images for local tasks is computationally expensive. To save memory, UMG-CLIP uses a cluster-based strategy that identifies and merges redundant visual tokens, reducing the workload by 75% without losing the spatial detail needed for local perception. Furthermore, for downstream tasks like segmentation, the model employs Parameter-Efficient Tuning (PET). By keeping the main model frozen and only training tiny "Convpass" modules, UMG-CLIP preserves its general knowledge while adapting cheaply to new tasks.
Datasets: Scaling Multi-Granularity
The core contribution to the training process is the UMG-41M dataset. It contains 41.41 million images with 389 million regional annotations across 11,741 classes. It was built by re-annotating six existing public datasets (CC3M, CC12M, SBU, VG, YFCC15M, and IN21k). Additionally, a specialized BigDet Panoptic dataset containing 3.59 million images was used for dense adaptation.
Key Results and Insights
UMG-CLIP consistently outperformed standard CLIP and its high-performing variants (like EVA-02-CLIP) across a wide spectrum of tasks.
- Zero-Shot Classification: UMG-CLIP-L/14 achieved an average top-1 accuracy of 72.1% across 21 benchmarks, beating the original OpenAI CLIP by 3.3%.
- Image Retrieval: On the COCO dataset, it saw a massive +5.3% improvement in Image-to-Text retrieval (R@1) compared to its closest competitors.
- Segmentation: It achieved state-of-the-art results in open-vocabulary panoptic segmentation, significantly outperforming specialized models like FC-CLIP, even while using fewer parameters.
Insights
- Pseudo-labels are Powerful: You don't need perfect human labels to reach state-of-the-art performance. If you have enough scale and a smart filtering process to remove low-quality "teacher" guesses, pseudo-annotations are sufficient to teach a model fine-grained localization.
- Clustering vs. Masking: Many models use random masking to speed up training. The authors found that for local understanding, random masking is destructive. Clustering similar tokens is a much better way to save compute while keeping the "map" of the image intact.
- The "Generalist" Advantage: By learning image-level, region-level, and pixel-level details simultaneously, the model becomes more robust. Improving its ability to see a "chair" in a small box actually helps it understand a "living room" scene better as a whole.
Conclusion
UMG-CLIP represents a significant advancement in vision-language foundation models by unifying global and local understanding. While the automated annotation process is currently resource-intensive, requiring multiple heavy teacher models, the results prove that multi-granularity alignment is key to creating true vision generalists. Future work will likely focus on making the data generation pipeline more efficient and mitigating noise in pseudo-labels.
The code is available at https://github.com/lygsbw/UMG-CLIP.
Summary updated