Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models

Taichi Iki, Akiko AizawaEMNLP 2021 · DOI

Introduction

Vision-and-Language (V&L) models are typically constructed by extending pre-trained language models (LMs) like BERT with visual processing capabilities. This "structural surgery" is followed by extensive pre-training on image-caption datasets. While the objective is to ground language in vision, there is a risk of degrading the model's original natural language understanding (NLU). This study investigates this trade-off, specifically comparing single-stream and dual-stream architectures across the GLUE benchmark.

Methodology

The study evaluates five V&L architectures:

  • Single-stream: VisualBERT, Uniter, and VL-BERT (where text and vision are processed in a single sequence).
  • Dual-stream: ViLBERT and LXMERT (where text and vision are processed in separate streams before fusion).

To isolate the effects of architecture and training, all models were initialized from BERT-base and pre-trained on Google’s Conceptual Captions (2.7M pairs). Evaluation was conducted on the GLUE benchmark. Since GLUE tasks are text-only, models were provided with "blank" black images to maintain the required input format without introducing external visual information.

Performance Degradation and Architectural Impact

Results indicate a consistent decline in NLU performance across all V&L models compared to the BERT-base baseline. While BERT-base achieved an average GLUE score of 77.3, V&L models averaged 71.6.

Comparison of score distributions for single-stream and dual-stream models against the BERT-base baseline.

Surprisingly, dual-stream models—despite having more parameters and being designed to preserve the language encoder—performed worse than single-stream models. VisualBERT (single-stream) was the top performer at 72.5, while LXMERT (dual-stream) was the lowest at 70.5. Experiments with "Mod-only" versions (structural changes without V&L pre-training) showed minimal performance loss, identifying the V&L pre-training phase as the primary driver of linguistic "forgetting."

The Role of Data Distribution and Vocabulary Overlap

The degradation is partly attributed to the shift in data distribution. Image captions are often simpler and more descriptive than the diverse text used to train BERT. Furthermore, the study identified a positive correlation between the performance on specific GLUE tasks and the degree of vocabulary overlap (Simpson coefficient) between the pre-training data and the task data.

Correlation between vocabulary overlap (Simpson coefficient) and normalized task scores across different GLUE datasets.

Conclusion

The integration of visual modalities into language models comes at a measurable cost to pure linguistic reasoning. Single-stream architectures appear more resilient to this degradation than dual-stream counterparts. Future research should focus on pre-training objectives that explicitly preserve NLU capabilities while achieving visual grounding, potentially using more sophisticated "null" visual inputs for language-only inference.

Summary updated

BibTeX
@inproceedings{iki2021effect,
  author = {Taichi Iki and Akiko Aizawa},
  title = {{Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.167},
}
Figure