Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision

Tzu-Jui Julius Wang, Jorma Laaksonen, Tomas Langer, Heikki Arponen, Tom E. Bishop

TL;DR

This paper introduces a novel "Visual Vocabulary based Feature Hallucinator" (WFH) to improve weakly-supervised vision-language pre-training (W-VLP). WFH generates visual representations from text, enabling richer cross-modal interactions and significantly boosting performance on tasks like cross-modal retrieval, especially in challenging cross-dataset scenarios.

What Problem Does This Paper Address?

Vision-language pre-training (VLP) aims to build models that understand the relationship between images and text. Traditionally, this requires large datasets of perfectly aligned image-caption pairs. However, collecting and annotating such data is expensive and time-consuming. Weakly-supervised VLP (W-VLP) attempts to overcome this by using less-than-perfectly aligned data, often relying on object tags extracted from images. While this approach has shown promise, the authors argue that relying solely on limited-semantics object tags can curb and bias the model's learning, particularly for tasks like cross-modal retrieval (XMR). They aim to enhance W-VLP by generating more diverse and semantically rich visual representations from text, even when paired data is scarce.

Methods

The core contribution is the Visual Vocabulary based Feature Hallucinator (WFH). WFH is trained alongside a W-VLP model and takes textual tokens as input. It uses a multi-layer Transformer architecture with self-attention and cross-attention mechanisms. The key idea is to "hallucinate" visual representations that correspond to the input text. This is achieved by attending to a pre-learned visual dictionary (derived from image features) based on the contextualized text embeddings. These generated visual features are then paired with the original, unpaired text for further pre-training. This process allows for more diverse interactions between modalities than relying solely on object tags. The overall W-VLP model then uses a combination of losses, including Masked Language Modeling (MLM), Masked Tag Classification (MTC), Masked Object Classification (MOC), and a new WFH loss, to learn from unpaired image-text data.

Datasets

  • Pre-training: Conceptual Captions (CC) dataset (2.7M unpaired image-caption pairs).
  • Fine-tuning/Evaluation:
    • XMR: MSCOCO [33] and Flickr30K [39].
  • VQA: VQAv2 [2, 14] and GQA [21].
  • VE: SNLI-VE [51].
  • REC: RefCOCO+ [55].

Key Insights and Ideas

  • Beyond Object Tags: The paper highlights the limitations of relying solely on object tags for W-VLP, arguing they offer insufficient semantic richness. WFH provides a way to generate more nuanced visual representations directly from text.
  • "Hallucinating" for Alignment: The core innovation is generating visual features from text. This "hallucination" process, guided by a visual dictionary, allows for richer cross-modal alignment without explicit paired data.
  • Improved Generalization: The proposed WFH model demonstrates significantly better generalization, particularly in cross-dataset retrieval tasks, suggesting that the generated features capture more robust and transferable representations.
  • Early Cross-Domain Alignment: Analysis shows that WFH models achieve stronger cross-domain alignments (specifically, language-to-vision cross-attention) in earlier Transformer layers compared to baselines, indicating a more efficient learning process.
  • Expressive Embeddings: Spectral analysis suggests that WFH leads to more expressive word token embeddings, which in turn aids in learning better cross-modal alignments.

Key Results

  • XMR Performance: WFH consistently improves upon the U-VisualBERT (U-VB) baseline on both Flickr30K and MSCOCO datasets, achieving gains in recall metrics (R@1, R@5, R@10).
  • Cross-Dataset Generalization: WFH shows substantial improvements in cross-dataset XMR tasks, with gains of up to 14.5% in R@1 on Flickr30K train to MSCOCO test, and 16.5% on MSCOCO train to Flickr30K test, compared to U-VB.
  • Other V-L Tasks: For VQA, VE, and REC tasks, WFH models achieve competitive results, often on par with models trained with fully paired data, and show notable improvements over U-VB on specific datasets like GQA and RefCOCO+.
  • Qualitative Findings: Visualizations suggest that hallucinated features are contextual and act as effective bridging representations between modalities.

Limitations and Future Work

  • Weak Supervision Nuance: The authors acknowledge that their "weakly-supervised" approach still relies on object detectors trained on paired data, meaning the models are not truly unsupervised. Future work aims to address this limitation to achieve fully unpaired vision-language pre-training.
  • Varying Supervision Levels: The paper suggests studying how WFH models generalize with varying amounts of supervision (e.g., different numbers of tags used).
  • Configuration Studies: Further studies on WFH configurations (e.g., attribute token usage, visual dictionary sizes) are mentioned as being available in supplementary material, implying these are areas for further exploration.

Summary updated

Abstract

Weakly-supervised vision-language (V-L) pre-training (W-VLP) aims at learning cross-modal alignment with little or no paired data, such as aligned images and captions. Recent W-VLP methods, which pair visual features with object tags, help achieve performances comparable with some VLP models trained with aligned pairs in various V-L downstream tasks. This, however, is not the case in cross-modal retrieval (XMR). We argue that the learning of such a W-VLP model is curbed and biased by the object tags of limited semantics.We address the lack of paired V-L data for model supervision with a novel Visual Vocabulary based Feature Hallucinator (WFH), which is trained via weak supervision as a W-VLP model, not requiring images paired with captions. WFH generates visual hallucinations from texts, which are then paired with the originally unpaired texts, allowing more diverse interactions across modalities.Empirically, WFH consistently boosts the prior W-VLP works, e.g. U-VisualBERT (U-VB), over a variety of V-L tasks, i.e. XMR, Visual Question Answering, etc. Notably, benchmarked with recall@{1,5,10}, it consistently U-VB on image-to-text and improves text-to-image retrieval on two popular datasets Flickr30K and MSCOCO. Meanwhile, it gains by at least 14.5% in cross-dataset generalization tests on these XMR tasks. Moreover, in other V-L downstream tasks considered, our WFH models are on par with models trained with paired V-L data, revealing the utility of unpaired data. These results demonstrate greater generalization of the proposed W-VLP model with WFH.
BibTeX
@inproceedings{wang2023learning,
  author = {Tzu-Jui Julius Wang and Jorma Laaksonen and Tomas Langer and Heikki Arponen and Tom E. Bishop},
  title = {{Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=1RsOxHUoka},
}
Figure