PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting

Zixin Guo, Tzu-Jui Julius Wang, Selen Pehlivan, Abduljalil Radman, Jorma Laaksonen

TL;DR

This paper introduces PiTL, a novel weakly-supervised vision-language pre-training (W-VLP) method that leverages large language models (LLMs) to generate descriptive text for image categories, significantly reducing the need for expensive, fully-aligned image-text pairs. PiTL achieves competitive and often superior performance in cross-modal retrieval tasks compared to existing methods, even with substantially less supervision.

What Problem Does This Paper Address?

Vision-language pre-training (VLP) has revolutionized tasks like cross-modal retrieval, but it typically relies on massive datasets of precisely aligned image-text pairs. Creating these datasets is costly and time-consuming. While weakly-supervised VLP (W-VLP) methods have emerged to reduce this burden, they often still require object-level annotations (e.g., bounding boxes) to train object detectors, which then provide tags for supervision. This paper tackles the challenge of further reducing supervision requirements for VLP, aiming for a method that uses only image-level category labels and general knowledge from LLMs to generate effective training data. The goal is to bridge the gap between fully supervised VLP and existing W-VLP methods, making VLP more accessible and data-efficient.

Methods

The core innovation is Prompts-in-The-Loop (PiTL). Instead of relying on object detectors or human-annotated captions, PiTL uses image category labels (e.g., "refinery") as a starting point. These labels are fed into a large language model (LLM), like GPT-3, along with carefully crafted prompts. These prompts are designed to elicit descriptive knowledge about the category, covering aspects like colors, shapes, scenes, and relations with other entities. For instance, a prompt might be "Describe what a <category> could be seen with?". The LLM's responses are then used as textual descriptions to create image-text pairs.

These generated pairs are then used to pre-train vision-language models. The authors employ a standard VLP architecture, similar to ALBEF, which includes visual and textual encoders followed by a fusion encoder. The pre-training objective comprises four losses: Image-Text Contrastive (ITC), Image-Text Matching (ITM), Masked Language Modeling (MLM), and Intra-Modal Contrastive (IMC). A key aspect of PiTL's training is that an image is associated with a group of descriptions from the same category, promoting category-level alignment rather than instance-level alignment.

Datasets

The authors introduce a new dataset called IN14K. This dataset comprises 9 million images and 1 million descriptions derived from the "Winter21" release of ImageNet-21K, using the PiTL methodology. They also created subsets IN1K and IN6K from this data for experiments. For evaluation, they used standard cross-modal retrieval benchmarks: MSCOCO-5K and Flickr30K.

Key Insights and Ideas

  • LLMs as Knowledge Extractors: The paper demonstrates that LLMs can effectively extract rich, descriptive knowledge from simple category labels, serving as a powerful substitute for human-annotated captions or object detectors.
  • Category-Level Alignment: By associating an image with multiple LLM-generated descriptions of the same category, PiTL encourages models to learn broader, categorical understanding rather than just instance-specific associations. This is crucial for generalization.
  • Reduced Supervision, High Performance: PiTL achieves state-of-the-art or competitive results in cross-modal retrieval with significantly less supervision than previous W-VLP methods, highlighting the efficiency of prompt-based knowledge generation.
  • Prompt Engineering Matters: Different types of prompts yield varying performance. Visually-oriented prompts (e.g., colors, shapes) tend to be better for text-to-image retrieval, while more open-ended prompts can be beneficial for image-to-text retrieval.

Key Results

  • PiTL models pre-trained on the new IN14K dataset (9M images, 1M descriptions) achieved state-of-the-art or highly competitive results on MSCOCO-5K and Flickr30K for both image-to-text (I2T) and text-to-image (T2I) retrieval.
  • For instance, on MSCOCO-5K, PiTL models outperformed existing W-VLP methods by significant margins, achieving an 11% and 10% improvement in I2T and T2I recall@1, respectively, when using the full IN14K dataset.
  • PiTL models pre-trained on IN14K were comparable to VLP models trained on much larger datasets (e.g., 4M aligned image-text pairs from CC3M and SBU Captions).
  • Ablation studies showed that using all nine types of prompts together yielded the best overall retrieval performance.

Limitations and Future Work

  • Prompt Sensitivity: The performance is influenced by the choice of prompts, suggesting that careful prompt engineering is necessary. The paper notes that while all prompts together are best, some individual prompts are better suited for specific retrieval directions (I2T vs. T2I).
  • LLM Dependence: The quality of generated descriptions is inherently tied to the capabilities of the LLM used.
  • Future Work: The authors suggest exploring more sophisticated prompt generation strategies and investigating how PiTL can be extended to other vision-language tasks beyond retrieval. They also highlight that PiTL significantly narrows the performance gap between W-VLP and fully supervised VLP.

Summary updated

Abstract

Vision-language (VL) Pre-training (VLP) has shown to well generalize VL models over a wide range of VL downstream tasks, especially for cross-modal retrieval. However, it hinges on a huge amount of image-text pairs, which requires tedious and costly curation. On the contrary, weakly-supervised VLP (W-VLP) explores means with object tags generated by a pre-trained object detector (OD) from images. Yet, they still require paired information, i.e. images and object-level annotations, as supervision to train an OD. To further reduce the amount of supervision, we propose Prompts-in-The-Loop (PiTL) that prompts knowledge from large language models (LLMs) to describe images. Concretely, given a category label of an image, e.g. refinery, the knowledge, e.g. a refinery could be seen with large storage tanks, pipework, and ..., extracted by LLMs is used as the language counterpart. The knowledge supplements, e.g. the common relations among entities most likely appearing in a scene. We create IN14K, a new VL dataset of 9M images and 1M descriptions of 14K categories from ImageNet21K with PiTL. Empirically, the VL models pre-trained with PiTL-generated pairs are strongly favored over other W-VLP works on image-to-text (I2T) and text-to-image (T2I) retrieval tasks, with less supervision. The results reveal the effectiveness of PiTL-generated pairs for VLP.
BibTeX
@inproceedings{guo2023pitl,
  author = {Zixin Guo and Tzu-Jui Julius Wang and Selen Pehlivan and Abduljalil Radman and Jorma Laaksonen},
  title = {{PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3592038},
}
Figure