CLIM: Contrastive Language-Image Mosaic for Region Representation

Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, Chen Change Loy

Abstract

CLIM (Contrastive Language-Image Mosaic) is a novel training strategy designed to bridge the gap between global image-text alignment and fine-grained region-text understanding in open-vocabulary object detection (OVD). By constructing mosaics of images and treating them as "pseudo regions," CLIM enables models to learn precise spatial-semantic associations without the need for expensive manual bounding box annotations. This approach significantly boosts the performance of existing detectors on novel, unseen categories.

Introduction: The Region Alignment Bottleneck

Traditional object detectors are constrained by fixed category sets. Open-vocabulary object detection (OVD) seeks to overcome this by allowing models to identify any object described in natural language. However, the primary bottleneck for OVD is region-text alignment—the ability to map specific visual areas to descriptive text.

Currently, models rely on either expensive human-labeled bounding boxes, which are slow to produce and difficult to scale, or noisy automatic labels generated by pre-trained models, which are often inaccurate. While massive datasets of image-text pairs (like those used for CLIP) exist, they provide only global alignment; they tell a model what is in an image, but not where it is located.

Methodology: Contrastive Language-Image Mosaic

CLIM introduces an elegantly simple solution to the localization problem by creating "pseudo regions" through image mosaics.

Comparison of region-text alignment strategies: (a) standard image-text pairs, (b) noisy region-text pairs from pre-trained detectors, and (c) CLIM's pseudo region-text pairs.

The Mosaic Strategy

Instead of training on individual images, CLIM constructs a grid-based mosaic (e.g., 2x2, 3x3, or 4x4) where each cell is filled with a different image from a standard dataset. Because the grid is constructed programmatically, the exact coordinates of each sub-image are known by design. These sub-images act as "pseudo regions" within the larger mosaic canvas.

Contrastive Learning Mechanism

The training process involves two primary encoders:

  1. Vision Encoder: Processes the entire mosaic to produce a unified feature map.
  2. Text Encoder: Processes the original captions corresponding to each sub-image in the mosaic.
The CLIM training process: pseudo regions in a mosaic are aligned with their corresponding text descriptions using a contrastive "Pull" and "Push" loss on the feature map.

The model is trained using a contrastive loss (InfoNCE). It learns to "pull" the feature representation of a specific pseudo region toward its corresponding text embedding while "pushing" it away from the embeddings of other images in the mosaic. This forces the model to learn localized, discriminative representations within a shared feature space without ever seeing a human-drawn bounding box.

Key Insights and Ideas

  • Alignment "For Free": CLIM bypasses the need for manual labels by using the mosaic structure to provide ground-truth locations by construction.
  • Orthogonality: CLIM is a training paradigm rather than a specific architecture. It can be integrated into existing state-of-the-art OVD frameworks like Detic and BARON, improving their performance regardless of their internal logic.
  • Resolution and Context: Higher resolution mosaics (e.g., 1024x1024) yield better results as they allow the model to capture finer details. Furthermore, training on a mosaic forces the model to distinguish between different concepts within a single feature map, enhancing contextual awareness.

Experimental Results

CLIM consistently outperforms baseline models across standard benchmarks:

  • Detic Enhancement: On the OV-COCO benchmark, adding CLIM to the Detic model improved performance on novel categories from 30.3 to 35.4 AP₅₀, a significant +5.1 gain.
  • OV-LVIS Success: On the larger LVIS dataset (1,200+ categories), CLIM improved Detic’s rare category mask mAP by 2.3 points.
  • Zero-Shot Accuracy: When applied to a standard CLIP (ViT-B-16) backbone, CLIM improved zero-shot region classification accuracy on COCO from 29.2% to 62.2%.

Conclusion and Future Work

CLIM demonstrates that structured data augmentation can effectively substitute for expensive manual labeling in region-level tasks. While the method is computationally intensive at high resolutions and currently limited by its rectangular grid bias, it provides a scalable and effective path for improving open-vocabulary understanding. Future work may explore more complex, non-grid collaging techniques to better represent the irregular shapes and overlaps found in real-world scenes.

Summary updated

Abstract

Detecting objects accurately from a large or open vocabulary necessitates the vision-language alignment on region representations. However, learning such a region-text alignment by obtaining high-quality box annotations with text labels or descriptions is expensive and infeasible. In contrast, collecting image-text pairs is simpler but lacks precise object location information to associate regions with texts. In this paper, we propose a novel approach called Contrastive Language-Image Mosaic (CLIM), which leverages large-scale image-text pairs effectively for aligning region and text representations. CLIM combines multiple images into a mosaicked image and treats each image as a ‘pseudo region’. The feature of each pseudo region is extracted and trained to be similar to the corresponding text embedding while dissimilar from others by a contrastive loss, enabling the model to learn the region-text alignment without costly box annotations. As a generally applicable approach, CLIM consistently improves different open-vocabulary object detection methods that use caption supervision. Furthermore, CLIM can effectively enhance the region representation of vision-language models, thus providing stronger backbones for open-vocabulary object detectors. Our experimental results demonstrate that CLIM improves different baseline open-vocabulary object detectors by a large margin on both OV-COCO and OV-LVIS benchmarks. The code is available at https://github.com/wusize/CLIM.
BibTeX
@inproceedings{wu2024clim,
  author = {Size Wu and Wenwei Zhang and Lumin Xu and Sheng Jin and Wentao Liu and Chen Change Loy},
  title = {{CLIM: Contrastive Language-Image Mosaic for Region Representation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28428},
}
Figure