Abstract
VL-BERT is a pioneering "single-stream" Transformer architecture designed to learn generic visual-linguistic representations. By treating image regions and text tokens as equal parts of a unified sequence and pre-training on massive multi-modal and text-only datasets, VL-BERT achieves state-of-the-art performance across various downstream tasks, most notably the Visual Commonsense Reasoning (VCR) benchmark.
Introduction
In the fields of Computer Vision (CV) and Natural Language Processing (NLP), "backbone" models like ResNet and BERT have revolutionized task-specific performance through large-scale pre-training. However, Visual-Linguistic (VL) tasks, such as Visual Question Answering (VQA), have historically lacked a similar generic foundation. Previous approaches often relied on task-specific "Frankenstein" architectures that combined pre-trained CV and NLP models. These models frequently suffered from overfitting because cross-modal connections had to be learned from scratch on relatively small datasets. VL-BERT addresses this by providing a unified, pre-trainable backbone for multi-modal understanding.
Methods
VL-BERT adapts the Transformer architecture to process both visual and linguistic features simultaneously in a "single-stream" fashion.
Architecture
Unlike dual-stream models that process modalities separately before merging, VL-BERT treats each input element—whether a word piece or a Region-of-Interest (RoI)—as a token in a single sequence. Each input element is represented by the summation of four embeddings: Token, Visual Feature (including geometry), Segment, and Sequence Position.
Pre-training Objectives
The model is trained on two primary objectives to align modalities:
- Masked Language Modeling (MLM) with Visual Clues: Similar to BERT, the model predicts masked words using both surrounding text and unmasked image regions.
- Masked RoI Classification with Linguistic Clues: Image regions are masked (pixels zeroed), and the model predicts the object category using linguistic context and other visual regions.
Datasets and Evaluation
VL-BERT is pre-trained on the Conceptual Captions dataset (~3.3M images) and text-only corpora (BooksCorpus and Wikipedia). The inclusion of text-only data is crucial for maintaining linguistic robustness. Evaluation was conducted on major benchmarks including VCR (Visual Commonsense Reasoning), VQA v2.0, and RefCOCO+.
Key Results
VL-BERT demonstrated significant improvements over existing baselines. On the VCR benchmark, it achieved first place for a single model at the time of publication, significantly outperforming the R2C baseline. It also surpassed the BUTD baseline on VQA by over 5% and showed a substantial lead on RefCOCO+.
The success of VL-BERT highlights several key insights: the single-stream approach allows for early cross-modal interaction, and pre-training on text-only data prevents the model from losing complex grammatical understanding while learning visual features.
Conclusion and Future Work
VL-BERT establishes a powerful foundation for VL tasks. However, it remains dependent on pre-trained object detectors for RoI extraction. Future research may explore end-to-end "pixels-to-text" architectures and extending pre-training to generative tasks like image captioning to further enhance representation quality.
Summary updated