Abstract
To alleviate the cost of obtaining accurate bounding boxes for training today's state-of-the-art object detection models, recent weakly supervised detection work has proposed techniques to learn from image-level labels. However, requiring discrete image-level labels is both restrictive and suboptimal. Real-world "supervision" usually consists of more unstructured text, such as captions. In this work we learn association maps between images and captions. We then use a novel objectness criterion to rank the resulting candidate boxes, such that high-ranking boxes have strong gradients along all edges. Thus, we can detect objects beyond a fixed object category vocabulary, if those objects are frequent and distinctive enough. We show that our objectness criterion improves the proposed bounding boxes in relation to prior weakly supervised detection methods. Further, we show encouraging results on object detection from image-level captions only.
Introduction
Standard object detection typically operates in a "closed-world" environment, where models are trained to identify a fixed set of categories—such as the 80 classes in the COCO dataset—using thousands of manually annotated bounding boxes. This reliance on dense supervision is prohibitively expensive to scale. Weakly Supervised Object Detection (WSOD) attempts to mitigate this by training on image-level labels. However, even WSOD is constrained by predefined vocabularies.
In contrast, the real world offers abundant, unstructured supervision in the form of captions. The challenge lies in the noisy nature of this text, which often includes abstract terms, references to multiple instances, or mentions of objects not visually present. This paper introduces a method to leverage this "labor-free" text to discover and localize objects in an open-vocabulary setting.
Methods
The proposed approach utilizes a three-stage pipeline to bridge the gap between unstructured text and spatial object localization.
1. Multi-modal Association
The model learns a shared embedding space for image regions and words. Using a CNN for visual features and GloVe for word embeddings, a triplet loss is employed to ensure that an image and its corresponding caption have higher similarity scores than mismatched pairs. This process generates Class Activation Maps (CAMs), which highlight the pixels most relevant to specific words in the caption.
2. Generating Bounding Boxes with Objectness
To convert fuzzy heatmaps into precise bounding boxes, the authors introduce a novel objectness criterion. Unlike traditional methods that prioritize high average activation, this criterion requires a box to exhibit strong gradients along all four edges. A box is considered high-quality if the activation drops sharply at its boundaries, suggesting a tight fit around a distinct object.
3. Training the Detector
Finally, a modified Faster-RCNN is trained. To handle caption noise, the model factorizes the prediction into separate "objectness" and "classification" components. Multiple Instance Learning (MIL) is used to manage the uncertainty regarding which proposal box correctly matches a specific word in the caption.
Key Insights and Ideas
A core technical contribution is the "Four-Edge" Rule. By penalizing boxes that lack sharp activation drops on all sides, the model avoids over-merging objects with the background or adjacent items.
Furthermore, because the model learns directly from raw text, it can discover objects outside standard datasets. It successfully localizes novel categories like "tree stump," "table," "bridge," or "baby" simply because these terms frequently appear in captions alongside consistent visual patterns.
Key Results
The effectiveness of the proposed method was evaluated on MS COCO and Pascal VOC 2007.
- Proposal Quality: The edge-based objectness metric significantly outperformed baselines like EdgeBoxes in generating precise pseudo-ground-truth boxes.
- Detection Performance: On COCO, the model achieved 7% mAP. While lower than fully supervised counterparts, this represents a significant milestone for a model trained exclusively on noisy captions without human-drawn boxes.
- Transferability: When trained on COCO captions and tested on Pascal VOC 2007, the model achieved 24.8% mAP, demonstrating that the visual concepts learned from captions are robust and transferable across different benchmarks.
Conclusion
While the model shows promise, it faces challenges due to the semantic gap in captions. Abstract words like "elaborate" can confuse the localization process. Additionally, the current evaluation relies on hard string matching, which may penalize the model for using more descriptive synonyms than the ground truth labels. Future research will focus on integrating sophisticated language models to handle hierarchical relationships and synonyms, further refining the fusion of semantic activation and edge-based cues.
Summary updated