TL;DR
This paper introduces PiTL, a novel weakly-supervised vision-language pre-training (W-VLP) method that leverages large language models (LLMs) to generate descriptive text for image categories, significantly reducing the need for expensive, fully-aligned image-text pairs. PiTL achieves competitive and often superior performance in cross-modal retrieval tasks compared to existing methods, even with substantially less supervision.
What Problem Does This Paper Address?
Vision-language pre-training (VLP) has revolutionized tasks like cross-modal retrieval, but it typically relies on massive datasets of precisely aligned image-text pairs. Creating these datasets is costly and time-consuming. While weakly-supervised VLP (W-VLP) methods have emerged to reduce this burden, they often still require object-level annotations (e.g., bounding boxes) to train object detectors, which then provide tags for supervision. This paper tackles the challenge of further reducing supervision requirements for VLP, aiming for a method that uses only image-level category labels and general knowledge from LLMs to generate effective training data. The goal is to bridge the gap between fully supervised VLP and existing W-VLP methods, making VLP more accessible and data-efficient.
Methods
The core innovation is Prompts-in-The-Loop (PiTL). Instead of relying on object detectors or human-annotated captions, PiTL uses image category labels (e.g., "refinery") as a starting point. These labels are fed into a large language model (LLM), like GPT-3, along with carefully crafted prompts. These prompts are designed to elicit descriptive knowledge about the category, covering aspects like colors, shapes, scenes, and relations with other entities. For instance, a prompt might be "Describe what a <category> could be seen with?". The LLM's responses are then used as textual descriptions to create image-text pairs.
These generated pairs are then used to pre-train vision-language models. The authors employ a standard VLP architecture, similar to ALBEF, which includes visual and textual encoders followed by a fusion encoder. The pre-training objective comprises four losses: Image-Text Contrastive (ITC), Image-Text Matching (ITM), Masked Language Modeling (MLM), and Intra-Modal Contrastive (IMC). A key aspect of PiTL's training is that an image is associated with a group of descriptions from the same category, promoting category-level alignment rather than instance-level alignment.
Datasets
The authors introduce a new dataset called IN14K. This dataset comprises 9 million images and 1 million descriptions derived from the "Winter21" release of ImageNet-21K, using the PiTL methodology. They also created subsets IN1K and IN6K from this data for experiments. For evaluation, they used standard cross-modal retrieval benchmarks: MSCOCO-5K and Flickr30K.
Key Insights and Ideas
- LLMs as Knowledge Extractors: The paper demonstrates that LLMs can effectively extract rich, descriptive knowledge from simple category labels, serving as a powerful substitute for human-annotated captions or object detectors.
- Category-Level Alignment: By associating an image with multiple LLM-generated descriptions of the same category, PiTL encourages models to learn broader, categorical understanding rather than just instance-specific associations. This is crucial for generalization.
- Reduced Supervision, High Performance: PiTL achieves state-of-the-art or competitive results in cross-modal retrieval with significantly less supervision than previous W-VLP methods, highlighting the efficiency of prompt-based knowledge generation.
- Prompt Engineering Matters: Different types of prompts yield varying performance. Visually-oriented prompts (e.g., colors, shapes) tend to be better for text-to-image retrieval, while more open-ended prompts can be beneficial for image-to-text retrieval.
Key Results
- PiTL models pre-trained on the new IN14K dataset (9M images, 1M descriptions) achieved state-of-the-art or highly competitive results on MSCOCO-5K and Flickr30K for both image-to-text (I2T) and text-to-image (T2I) retrieval.
- For instance, on MSCOCO-5K, PiTL models outperformed existing W-VLP methods by significant margins, achieving an 11% and 10% improvement in I2T and T2I recall@1, respectively, when using the full IN14K dataset.
- PiTL models pre-trained on IN14K were comparable to VLP models trained on much larger datasets (e.g., 4M aligned image-text pairs from CC3M and SBU Captions).
- Ablation studies showed that using all nine types of prompts together yielded the best overall retrieval performance.
Limitations and Future Work
- Prompt Sensitivity: The performance is influenced by the choice of prompts, suggesting that careful prompt engineering is necessary. The paper notes that while all prompts together are best, some individual prompts are better suited for specific retrieval directions (I2T vs. T2I).
- LLM Dependence: The quality of generated descriptions is inherently tied to the capabilities of the LLM used.
- Future Work: The authors suggest exploring more sophisticated prompt generation strategies and investigating how PiTL can be extended to other vision-language tasks beyond retrieval. They also highlight that PiTL significantly narrows the performance gap between W-VLP and fully supervised VLP.
Summary updated