VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai

Abstract

VL-BERT is a pioneering "single-stream" Transformer architecture designed to learn generic visual-linguistic representations. By treating image regions and text tokens as equal parts of a unified sequence and pre-training on massive multi-modal and text-only datasets, VL-BERT achieves state-of-the-art performance across various downstream tasks, most notably the Visual Commonsense Reasoning (VCR) benchmark.

Introduction

In the fields of Computer Vision (CV) and Natural Language Processing (NLP), "backbone" models like ResNet and BERT have revolutionized task-specific performance through large-scale pre-training. However, Visual-Linguistic (VL) tasks, such as Visual Question Answering (VQA), have historically lacked a similar generic foundation. Previous approaches often relied on task-specific "Frankenstein" architectures that combined pre-trained CV and NLP models. These models frequently suffered from overfitting because cross-modal connections had to be learned from scratch on relatively small datasets. VL-BERT addresses this by providing a unified, pre-trainable backbone for multi-modal understanding.

Methods

VL-BERT adapts the Transformer architecture to process both visual and linguistic features simultaneously in a "single-stream" fashion.

Architecture

Unlike dual-stream models that process modalities separately before merging, VL-BERT treats each input element—whether a word piece or a Region-of-Interest (RoI)—as a token in a single sequence. Each input element is represented by the summation of four embeddings: Token, Visual Feature (including geometry), Segment, and Sequence Position.

Overview of the VL-BERT architecture and its pre-training objectives, including Masked Language Modeling and Masked RoI Classification.

Pre-training Objectives

The model is trained on two primary objectives to align modalities:

  • Masked Language Modeling (MLM) with Visual Clues: Similar to BERT, the model predicts masked words using both surrounding text and unmasked image regions.
  • Masked RoI Classification with Linguistic Clues: Image regions are masked (pixels zeroed), and the model predicts the object category using linguistic context and other visual regions.

Datasets and Evaluation

VL-BERT is pre-trained on the Conceptual Captions dataset (~3.3M images) and text-only corpora (BooksCorpus and Wikipedia). The inclusion of text-only data is crucial for maintaining linguistic robustness. Evaluation was conducted on major benchmarks including VCR (Visual Commonsense Reasoning), VQA v2.0, and RefCOCO+.

Key Results

VL-BERT demonstrated significant improvements over existing baselines. On the VCR benchmark, it achieved first place for a single model at the time of publication, significantly outperforming the R2C baseline. It also surpassed the BUTD baseline on VQA by over 5% and showed a substantial lead on RefCOCO+.

Performance comparison on the VCR benchmark showing VL-BERT's superiority over previous models like R2C and ViLBERT.

The success of VL-BERT highlights several key insights: the single-stream approach allows for early cross-modal interaction, and pre-training on text-only data prevents the model from losing complex grammatical understanding while learning visual features.

Conclusion and Future Work

VL-BERT establishes a powerful foundation for VL tasks. However, it remains dependent on pre-trained object detectors for RoI extraction. Future research may explore end-to-end "pixels-to-text" architectures and extending pre-training to generative tasks like image captioning to further enhance representation quality.

Summary updated

Abstract

VL-BERT is a simple yet powerful pre-trainable generic representation for visual-linguistic tasks. It is pre-trained on the massive-scale caption dataset and text-only corpus, and can be finetuned for varies down-stream visual-linguistic tasks.
BibTeX
@inproceedings{su2020vl,
  author = {Weijie Su and Xizhou Zhu and Yue Cao and Bin Li and Lewei Lu and Furu Wei and Jifeng Dai},
  title = {{VL-BERT: Pre-training of Generic Visual-Linguistic Representations}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure