Abstract
Open-vocabulary object detection (OVD) requires models to align text descriptions with specific image regions. However, obtaining high-quality region-level annotations is prohibitively expensive. We present CLIM (Contrastive Language-Image Mosaic), a framework that leverages abundant image-wide captions to learn localized visual concepts. By constructing synthetic "mosaics" where each sub-image acts as a known pseudo region, CLIM provides precise spatial supervision without manual bounding boxes. Our approach significantly improves performance across multiple OVD benchmarks and architectures.
Introduction: The Challenge of Spatial Precision
Modern computer vision is moving toward Open-Vocabulary Object Detection (OVD)—the ability to detect any object described in text, even if it wasn't in the training set. To do this well, a model must align specific image regions with specific words (e.g., knowing exactly which pixels represent a "vintage toaster").
The challenge lies in the data. High-quality datasets with manual bounding boxes and text descriptions are incredibly expensive to produce. Conversely, image-text pairs (like those used to train CLIP) are abundant but lack spatial precision; the model knows a "dog" is in the image, but not exactly where. Previous attempts to bridge this gap often relied on "noisy" region-text matching, using pre-trained proposal networks that are themselves prone to error.
Methodology: Contrastive Language-Image Mosaic
The core innovation of CLIM is the creation of a synthetic training environment where spatial locations are known by design. Instead of trying to guess where objects are in a single image, the authors create a synthetic image where the locations are known by design:
- Mosaicking: At each training step, a large canvas is divided into a grid (e.g., $2 \times 2$, $3 \times 3$, or $4 \times 4$).
- Pseudo Regions: Each cell in the grid is filled with a randomly sampled image from a standard image-text dataset. These sub-images are treated as "pseudo regions."
- Contrastive Alignment: The entire mosaic is passed through a vision encoder. Features are extracted from the specific grid locations. Simultaneously, the original captions for each sub-image are passed through a text encoder.
- The Loss: A contrastive loss (InfoNCE) is applied to pull the feature of a pseudo region toward its corresponding text embedding while pushing it away from all other captions in the mosaic.
By construction, the model knows exactly where each "pseudo region" starts and ends. This eliminates the need for a Region Proposal Network (RPN) during the alignment phase, avoiding the "noisy" matching problem of previous methods.
Key Insights
- Forced Local Attention: When a model sees a mosaic, it cannot rely on global image statistics to match a caption. It is forced to learn the distinct visual features within a specific spatial boundary to satisfy the contrastive loss.
- Scale and Diversity: Using a randomized grid size ensures the model learns to recognize concepts at various scales, improving its generalization to real-world objects of different sizes.
- Orthogonality: CLIM isn't a replacement for existing OVD methods; it's an enhancement. It works alongside techniques like image-tagging or pseudo-labeling to provide a stronger foundation for region-text alignment.
Key Results
The quantitative improvements are substantial across multiple architectures:
- Detic + CLIM: On the OV-COCO benchmark, CLIM improved the detection of novel categories by 5.1 AP₅₀ (from 30.3 to 35.4).
- BARON + CLIM: Saw a boost of 2.1 AP₅₀ on novel categories.
- OV-LVIS: CLIM improved the Detic baseline by 2.3 mAP and the F-VLM model by 2.2 mAP on rare categories.
- Qualitative Success: Feature map visualizations show that CLIM-trained models produce much sharper, more localized responses to text queries. While a standard CLIP model might have a "diffused" response to a query like "laptop," a CLIM-enhanced model pinpoints the object accurately.
Limitations and Future Work
- Computational Cost: Training on high-resolution mosaics (e.g., $1024 \times 1024$) yields the best results but increases computational overhead.
- Synthetic Context: A pseudo region's "background" in a mosaic is other random images, which is not natural. While this forces the model to focus on the region itself, it might miss out on learning how objects relate to their natural surroundings.
- Future Directions: The authors suggest exploring "composed regions"—merging adjacent grid cells to create rectangular pseudo regions—to better represent objects with extreme aspect ratios.
Summary updated