Abstract
Computer vision tasks such as object detection and semantic/instance segmentation rely on the painstaking annotation of large training datasets. In this paper, we propose LocTex that takes advantage of the low-cost localized textual annotations (i.e., captions and synchronized mouse-over gestures) to reduce the annotation effort. We introduce a contrastive pre-training framework between images and captions and propose to supervise the cross-modal attention map with rendered mouse traces to provide coarse localization signals. Our learned visual features capture rich semantics (from free-form captions) and accurate localization (from mouse traces), which are very effective when transferred to various downstream vision tasks. Compared with ImageNet supervised pre-training, LocTex can reduce the size of the pre-training dataset by 10x or the target dataset by 2x while achieving comparable or even improved performance on COCO instance segmentation. When provided with the same amount of annotations, LocTex achieves around 4% higher accuracy than the previous state-of-the-art "vision+language" pre-training approach on the task of PASCAL VOC image classification.
Introduction
The standard "pre-train then fine-tune" paradigm in computer vision typically relies on large-scale supervised pre-training on datasets like ImageNet. However, this approach faces two significant challenges: the high cost of fine-grained classification labeling and a "localization gap," where classification-focused pre-training encourages spatial invariance, which is detrimental to downstream tasks like object detection or instance segmentation.
LocTex addresses these issues by leveraging "localized narratives"—captions synchronized with mouse traces. This provides a low-cost, human-centric signal that teaches models both "what" an object is and "where" it is located.
Methodology: The LocTex Framework
LocTex bridges vision and language modalities through two primary objectives:
- Contrastive Learning: Using a ResNet-50 visual backbone and a Transformer textual backbone, the model employs an InfoNCE loss to align matching image-caption pairs in a shared feature space. This captures global semantic information.
- Localization Supervision: The core innovation of LocTex is supervising the cross-modal attention map between visual features and caption tokens using rendered mouse traces. By ensuring that the "cat" token attends to the pixels where the annotator's mouse moved while saying "cat," the model learns precise spatial grounding.
The authors found that supervising at a lower resolution (e.g., 7x7 or 14x14) is more effective, as it accommodates the inherent noise and shakiness of human mouse traces.
Experimental Evaluation
The learned representations were evaluated on PASCAL VOC and COCO. Key findings include:
- Data Efficiency: LocTex pre-trained on 118K images matches the performance of ImageNet pre-training on 1.28M images for COCO instance segmentation—a 10x reduction in data requirements.
- Superior Accuracy: Compared to previous vision+language methods like VirTex, LocTex achieves 4% higher accuracy on PASCAL VOC classification given the same annotation effort.
- Scalability: Increasing the pre-training data to 809K images from Open Images further boosts performance, reaching 92.6% mAP on VOC classification.
Conclusion
LocTex demonstrates that "noisy" human gestures like mouse traces are powerful signals for learning spatially-aware visual representations. By combining contrastive learning with localization supervision, LocTex significantly improves data efficiency and downstream task performance. Future work includes exploring zero-shot transfer and using speech recognition to further lower annotation costs.
Summary updated