Introduction
Vision-and-Language (V&L) models are typically constructed by extending pre-trained language models (LMs) like BERT with visual processing capabilities. This "structural surgery" is followed by extensive pre-training on image-caption datasets. While the objective is to ground language in vision, there is a risk of degrading the model's original natural language understanding (NLU). This study investigates this trade-off, specifically comparing single-stream and dual-stream architectures across the GLUE benchmark.
Methodology
The study evaluates five V&L architectures:
- Single-stream: VisualBERT, Uniter, and VL-BERT (where text and vision are processed in a single sequence).
- Dual-stream: ViLBERT and LXMERT (where text and vision are processed in separate streams before fusion).
To isolate the effects of architecture and training, all models were initialized from BERT-base and pre-trained on Google’s Conceptual Captions (2.7M pairs). Evaluation was conducted on the GLUE benchmark. Since GLUE tasks are text-only, models were provided with "blank" black images to maintain the required input format without introducing external visual information.
Performance Degradation and Architectural Impact
Results indicate a consistent decline in NLU performance across all V&L models compared to the BERT-base baseline. While BERT-base achieved an average GLUE score of 77.3, V&L models averaged 71.6.
Surprisingly, dual-stream models—despite having more parameters and being designed to preserve the language encoder—performed worse than single-stream models. VisualBERT (single-stream) was the top performer at 72.5, while LXMERT (dual-stream) was the lowest at 70.5. Experiments with "Mod-only" versions (structural changes without V&L pre-training) showed minimal performance loss, identifying the V&L pre-training phase as the primary driver of linguistic "forgetting."
The Role of Data Distribution and Vocabulary Overlap
The degradation is partly attributed to the shift in data distribution. Image captions are often simpler and more descriptive than the diverse text used to train BERT. Furthermore, the study identified a positive correlation between the performance on specific GLUE tasks and the degree of vocabulary overlap (Simpson coefficient) between the pre-training data and the task data.
Conclusion
The integration of visual modalities into language models comes at a measurable cost to pure linguistic reasoning. Single-stream architectures appear more resilient to this degradation than dual-stream counterparts. Future research should focus on pre-training objectives that explicitly preserve NLU capabilities while achieving visual grounding, potentially using more sophisticated "null" visual inputs for language-only inference.
Summary updated