ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee

Abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, processing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

Introduction

ViLBERT represents a landmark shift in how AI models handle tasks that combine sight and sound. By adapting the revolutionary BERT architecture to a multi-modal setting, the authors demonstrate that "visual grounding"—the ability to link words to image content—is a capability that can be pretrained on massive datasets and then transferred to a wide variety of specific applications.

The Challenge of Visual Grounding

Traditionally, vision-and-language (V&L) models relied on a "pretrain-then-transfer" approach for each modality individually. You might use a ResNet pretrained on ImageNet for vision and a BERT model pretrained on Wikipedia for text.

The "gap" was the connection between them. Grounding (e.g., knowing that the word "beagle" refers to a specific set of pixels in an image) was usually learned from scratch for every new task. This is inefficient and often leads to models that "overfit" to the biases of small datasets rather than learning true semantic connections. ViLBERT asks: Can we pretrain the connection between vision and language itself?

Methods

The core innovation is the two-stream architecture. While some concurrent works tried to shove image regions and words into a single BERT sequence, ViLBERT keeps them in parallel tracks.

The ViLBERT two-stream architecture showing visual and linguistic processing paths.

Two-Stream Processing

The authors argue that images and text require different levels of processing. Visual features from an object detector are already high-level, while words in a sentence need significant context-aggregation. The two-stream approach allows for different network depths for each modality, avoiding "corrupting" the pretrained weights of the language model with raw visual data too early.

Co-Attentional Transformer Layers (Co-TRM)

This is the "Aha!" architectural tweak. In a standard Transformer, "self-attention" looks at other elements in the same stream. In a Co-TRM layer, the model exchanges keys and values between the visual and linguistic streams. This forces the text stream to attend to the image regions and vice versa, creating a truly joint representation.

Comparison between a standard Transformer (left) and the novel Co-Attentional Transformer (right).

Pretraining Tasks

To teach the model grounding, they use two "proxy" tasks on the Conceptual Captions dataset (~3.3 million pairs):

  • Masked Multi-modal Modeling: The model must predict a missing word or a missing image region (semantic class) based on the surrounding context from both modalities.
  • Multi-modal Alignment Prediction: The model is shown an image and a caption and must decide if they actually match.

Key Results

ViLBERT set new state-of-the-art (SOTA) benchmarks across all four evaluation tasks:

  • VQA 2.0: Improved performance to 70.55.
  • VCR (Visual Commonsense Reasoning): Significant jumps in reasoning (Q→AR) from 44.0 to 54.8.
  • Image Retrieval: On Flickr30k, it outperformed previous bests by a massive margin in zero-shot settings.
  • Ablation: The authors showed that pretraining on Conceptual Captions provided a 2-13% boost across tasks compared to using the same architecture without multi-modal pretraining.

Conclusion

ViLBERT shows that "understanding how English relates to pixels" is a general skill that can be learned once and used everywhere. While computationally expensive and currently limited to static images, it represents a significant step toward task-agnostic visiolinguistic representations.

Summary updated

Abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.
BibTeX
@inproceedings{lu2019vilbert,
  author = {Jiasen Lu and Dhruv Batra and Devi Parikh and Stefan Lee},
  title = {{ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=RbCjYByOgbp},
}
Figure