Abstract
CLIM (Contrastive Language-Image Mosaic) is a novel training strategy designed to bridge the gap between global image-text alignment and fine-grained region-text understanding in open-vocabulary object detection (OVD). By constructing mosaics of images and treating them as "pseudo regions," CLIM enables models to learn precise spatial-semantic associations without the need for expensive manual bounding box annotations. This approach significantly boosts the performance of existing detectors on novel, unseen categories.
Introduction: The Region Alignment Bottleneck
Traditional object detectors are constrained by fixed category sets. Open-vocabulary object detection (OVD) seeks to overcome this by allowing models to identify any object described in natural language. However, the primary bottleneck for OVD is region-text alignment—the ability to map specific visual areas to descriptive text.
Currently, models rely on either expensive human-labeled bounding boxes, which are slow to produce and difficult to scale, or noisy automatic labels generated by pre-trained models, which are often inaccurate. While massive datasets of image-text pairs (like those used for CLIP) exist, they provide only global alignment; they tell a model what is in an image, but not where it is located.
Methodology: Contrastive Language-Image Mosaic
CLIM introduces an elegantly simple solution to the localization problem by creating "pseudo regions" through image mosaics.
The Mosaic Strategy
Instead of training on individual images, CLIM constructs a grid-based mosaic (e.g., 2x2, 3x3, or 4x4) where each cell is filled with a different image from a standard dataset. Because the grid is constructed programmatically, the exact coordinates of each sub-image are known by design. These sub-images act as "pseudo regions" within the larger mosaic canvas.
Contrastive Learning Mechanism
The training process involves two primary encoders:
- Vision Encoder: Processes the entire mosaic to produce a unified feature map.
- Text Encoder: Processes the original captions corresponding to each sub-image in the mosaic.
The model is trained using a contrastive loss (InfoNCE). It learns to "pull" the feature representation of a specific pseudo region toward its corresponding text embedding while "pushing" it away from the embeddings of other images in the mosaic. This forces the model to learn localized, discriminative representations within a shared feature space without ever seeing a human-drawn bounding box.
Key Insights and Ideas
- Alignment "For Free": CLIM bypasses the need for manual labels by using the mosaic structure to provide ground-truth locations by construction.
- Orthogonality: CLIM is a training paradigm rather than a specific architecture. It can be integrated into existing state-of-the-art OVD frameworks like Detic and BARON, improving their performance regardless of their internal logic.
- Resolution and Context: Higher resolution mosaics (e.g., 1024x1024) yield better results as they allow the model to capture finer details. Furthermore, training on a mosaic forces the model to distinguish between different concepts within a single feature map, enhancing contextual awareness.
Experimental Results
CLIM consistently outperforms baseline models across standard benchmarks:
- Detic Enhancement: On the OV-COCO benchmark, adding CLIM to the Detic model improved performance on novel categories from 30.3 to 35.4 AP₅₀, a significant +5.1 gain.
- OV-LVIS Success: On the larger LVIS dataset (1,200+ categories), CLIM improved Detic’s rare category mask mAP by 2.3 points.
- Zero-Shot Accuracy: When applied to a standard CLIP (ViT-B-16) backbone, CLIM improved zero-shot region classification accuracy on COCO from 29.2% to 62.2%.
Conclusion and Future Work
CLIM demonstrates that structured data augmentation can effectively substitute for expensive manual labeling in region-level tasks. While the method is computationally intensive at high resolutions and currently limited by its rectangular grid bias, it provides a scalable and effective path for improving open-vocabulary understanding. Future work may explore more complex, non-grid collaging techniques to better represent the irregular shapes and overlaps found in real-world scenes.
Summary updated