ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee

The Vision-and-Language Bottleneck

In the distinct fields of Computer Vision (CV) and Natural Language Processing (NLP), the standard recipe for success is well-established: pretrain on massive datasets (like ImageNet or Wikipedia) and then fine-tune on specific downstream tasks. However, in the "Vision-and-Language" (V+L) sub-field—which involves tasks requiring joint understanding of images and text—this paradigm was historically absent.

Previously, researchers would take a pretrained vision model and a pretrained language model, attempting to "glue" them together during task-specific training. The fundamental flaw in this approach is that the model must learn how to link words to image regions (visual grounding) from scratch on relatively small, often biased datasets. This resulted in "myopic" models that performed well on specific datasets but failed to generalize.

ViLBERT (Vision-and-Language BERT) shifts this paradigm from task-specific training to a unified "pretrain-then-transfer" approach, creating a task-agnostic foundation for visual grounding.

Methods: The ViLBERT Architecture

To achieve generalized visual grounding, the authors extend the popular BERT architecture into a multi-modal two-stream model.

ViLBERT Two-Stream Architecture
  1. Two-Stream Processing: Unlike concurrent models that force image regions and words into a single sequence, ViLBERT processes them in parallel streams. This design choice allows each modality to have different network depths and processing styles. This is crucial because image features (extracted from deep CNNs) are already at a higher level of abstraction compared to raw text tokens.
  2. Image Representation: Images are treated as a sequence of regions. The model uses a pretrained Faster R-CNN to extract bounding boxes and their corresponding visual features, appending spatial information (coordinates and area) to help the network understand spatial relationships.

Co-Attentional Transformer Layers (Co-TRM)

The core innovation of ViLBERT is the Co-Attentional Transformer Layer.

Co-Attentional Transformer Layer

In a standard Transformer, a layer attends only to its own sequence. In a Co-TRM layer, the keys and values are swapped between the visual and linguistic streams. The visual stream "looks" at the text to decide which image regions are most relevant, while the linguistic stream "looks" at the image to ground the context of the words.

Pretraining Tasks

The model learns grounding through two proxy tasks on the Conceptual Captions dataset (~3.3 million web-scraped image-caption pairs):

  • Masked Multi-modal Modeling: Similar to BERT's masked language modeling, the model must predict the identity of masked words or the semantic category of masked image regions based on the surrounding multi-modal context.
  • Multi-modal Alignment Prediction: The model is presented with an image-caption pair and must predict whether they genuinely match or if the caption was randomly swapped.

Key Insights and Results

ViLBERT's approach yielded significant breakthroughs, setting new state-of-the-art (SOTA) marks on four major evaluation tasks at the time of publication:

  • VQA 2.0 (Visual Question Answering): Achieved a score of 70.55, a substantial leap over previous task-specific models.
  • VCR (Visual Commonsense Reasoning): Showed massive improvements, particularly in the "Question $\rightarrow$ Answer + Rationale" setting (54.04), where the model must justify its answer.
  • Zero-Shot Potential: Even without fine-tuning on specific retrieval datasets, the pretrained ViLBERT demonstrated a strong semantic understanding of image-text alignment, outperforming several baselines explicitly trained for that task.

Why does it work so well? The authors discovered that keeping the modalities separate but interacting (the two-stream approach) outperformed single-stream models. Forcing a model to treat image regions exactly like words can "damage" the rich linguistic patterns BERT has already learned. Furthermore, visual grounding—knowing that the word "beagle" refers to a specific four-legged object in the pixels—proved to be a fundamental, pretrainable skill that transfers beautifully across tasks.

Limitations and Future Directions

While groundbreaking, ViLBERT has a few notable limitations:

  • Generative Bottleneck: As a bidirectional model that processes the entire sequence at once, it is not naturally suited for text generation tasks like image captioning without specialized decoders.
  • Computational Cost & Dependency: The model relies heavily on a Faster R-CNN backbone to extract region features. This is computationally expensive and means the model's "vision" is strictly limited by the object detector's ability to find regions. If the detector misses an object, ViLBERT cannot "see" it.
  • Scope: The current architecture focuses on static images, leaving temporal grounding (video) and multi-turn grounded dialogues as open challenges for future work.

Summary updated

Abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, processing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.
BibTeX
@inproceedings{lu2019vilbert,
  author = {Jiasen Lu and Dhruv Batra and Devi Parikh and Stefan Lee},
  title = {{ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=RbCjYByOgbp},
}
Figure