TL;DR
This paper introduces a novel "Visual Vocabulary based Feature Hallucinator" (WFH) to improve weakly-supervised vision-language pre-training (W-VLP). WFH generates visual representations from text, enabling richer cross-modal interactions and significantly boosting performance on tasks like cross-modal retrieval, especially in challenging cross-dataset scenarios.
What Problem Does This Paper Address?
Vision-language pre-training (VLP) aims to build models that understand the relationship between images and text. Traditionally, this requires large datasets of perfectly aligned image-caption pairs. However, collecting and annotating such data is expensive and time-consuming. Weakly-supervised VLP (W-VLP) attempts to overcome this by using less-than-perfectly aligned data, often relying on object tags extracted from images. While this approach has shown promise, the authors argue that relying solely on limited-semantics object tags can curb and bias the model's learning, particularly for tasks like cross-modal retrieval (XMR). They aim to enhance W-VLP by generating more diverse and semantically rich visual representations from text, even when paired data is scarce.
Methods
The core contribution is the Visual Vocabulary based Feature Hallucinator (WFH). WFH is trained alongside a W-VLP model and takes textual tokens as input. It uses a multi-layer Transformer architecture with self-attention and cross-attention mechanisms. The key idea is to "hallucinate" visual representations that correspond to the input text. This is achieved by attending to a pre-learned visual dictionary (derived from image features) based on the contextualized text embeddings. These generated visual features are then paired with the original, unpaired text for further pre-training. This process allows for more diverse interactions between modalities than relying solely on object tags. The overall W-VLP model then uses a combination of losses, including Masked Language Modeling (MLM), Masked Tag Classification (MTC), Masked Object Classification (MOC), and a new WFH loss, to learn from unpaired image-text data.
Datasets
- Pre-training: Conceptual Captions (CC) dataset (2.7M unpaired image-caption pairs).
- Fine-tuning/Evaluation:
- XMR: MSCOCO [33] and Flickr30K [39].
- VQA: VQAv2 [2, 14] and GQA [21].
- VE: SNLI-VE [51].
- REC: RefCOCO+ [55].
Key Insights and Ideas
- Beyond Object Tags: The paper highlights the limitations of relying solely on object tags for W-VLP, arguing they offer insufficient semantic richness. WFH provides a way to generate more nuanced visual representations directly from text.
- "Hallucinating" for Alignment: The core innovation is generating visual features from text. This "hallucination" process, guided by a visual dictionary, allows for richer cross-modal alignment without explicit paired data.
- Improved Generalization: The proposed WFH model demonstrates significantly better generalization, particularly in cross-dataset retrieval tasks, suggesting that the generated features capture more robust and transferable representations.
- Early Cross-Domain Alignment: Analysis shows that WFH models achieve stronger cross-domain alignments (specifically, language-to-vision cross-attention) in earlier Transformer layers compared to baselines, indicating a more efficient learning process.
- Expressive Embeddings: Spectral analysis suggests that WFH leads to more expressive word token embeddings, which in turn aids in learning better cross-modal alignments.
Key Results
- XMR Performance: WFH consistently improves upon the U-VisualBERT (U-VB) baseline on both Flickr30K and MSCOCO datasets, achieving gains in recall metrics (R@1, R@5, R@10).
- Cross-Dataset Generalization: WFH shows substantial improvements in cross-dataset XMR tasks, with gains of up to 14.5% in R@1 on Flickr30K train to MSCOCO test, and 16.5% on MSCOCO train to Flickr30K test, compared to U-VB.
- Other V-L Tasks: For VQA, VE, and REC tasks, WFH models achieve competitive results, often on par with models trained with fully paired data, and show notable improvements over U-VB on specific datasets like GQA and RefCOCO+.
- Qualitative Findings: Visualizations suggest that hallucinated features are contextual and act as effective bridging representations between modalities.
Limitations and Future Work
- Weak Supervision Nuance: The authors acknowledge that their "weakly-supervised" approach still relies on object detectors trained on paired data, meaning the models are not truly unsupervised. Future work aims to address this limitation to achieve fully unpaired vision-language pre-training.
- Varying Supervision Levels: The paper suggests studying how WFH models generalize with varying amounts of supervision (e.g., different numbers of tags used).
- Configuration Studies: Further studies on WFH configurations (e.g., attribute token usage, visual dictionary sizes) are mentioned as being available in supplementary material, implying these are areas for further exploration.
Summary updated