BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning

Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan

TL;DR

BridgeTower introduces "bridge layers" that connect multiple intermediate layers of pre-trained vision and language encoders directly to a cross-modal fusion encoder. This simple architectural change allows the model to leverage different levels of semantic information, achieving state-of-the-art performance on vision-language tasks while using significantly less pre-training data than competing models.

What Problem Does This Paper Address?

Most modern Vision-Language (VL) models follow a "Two-Tower" architecture: one tower for images (visual encoder) and one for text (textual encoder), followed by a cross-modal encoder that fuses the two.

The authors identify a major inefficiency in how these models currently operate. Existing models typically either:

  1. Use very lightweight encoders and do all the heavy lifting (extraction and fusion) in the cross-modal encoder, which is often less effective.
  2. Use powerful, deep, pre-trained encoders (like ViT for vision and RoBERTa for text) but only feed the very last layer's output into the fusion tower.

The problem with the second approach is that it ignores the rich hierarchy of information contained in the intermediate layers. Lower layers in neural networks often capture surface-level features (like edges in images or syntax in text), while higher layers capture more abstract semantic concepts. By only fusing the most abstract representations, models miss out on crucial "bottom-up" alignment opportunities, making it harder for the fusion tower to learn how the modalities truly relate.

Methods

The core innovation is the BridgeTower architecture. Instead of a single connection at the end of the uni-modal encoders, BridgeTower builds multiple "bridges."

  • Architecture: It uses a pre-trained CLIP-ViT-B/16 as the visual encoder and RoBERTa-BASE as the textual encoder. The cross-modal encoder is a 6-layer transformer using a co-attention mechanism.
  • Bridge Layers: The top 6 layers of both the visual and textual encoders are connected to each of the 6 layers of the cross-modal encoder via bridge layers.
  • Simplicity is Key: The authors experimented with complex bridge designs but found that the simplest version—a basic residual-style connection followed by Layer Normalization (LayerNorm(x + y))—performed the best. This adds almost negligible computational overhead (only about 18.4K additional parameters).
  • Pre-training: The model was pre-trained using standard objectives: Masked Language Modeling (MLM) and Image-Text Matching (ITM).
Figure 1: (a) – (d) are four categories of current TWO-TOWER vision-language models; (e) gives a brief illustration of the BRIDGETOWER architecture. VE, TE, and CE are short for the Visual Encoder, Textual Encoder, and Cross-modal Encoder, respectively. The height of each rectangle represents its relative computational cost. VE = TE indicates that the visual encoder and the textual encoder have the same or a similar number of parameters or computational costs. Illustration inspired by ViLT.

Datasets

  • Pre-training: The model was pre-trained on a standard aggregate of 4 million unique images from four datasets: Conceptual Captions (CC), SBU Captions, MSCOCO, and Visual Genome (VG).
  • Evaluation: The authors evaluated BridgeTower on a wide array of downstream VL benchmarks:
  • VQAv2: Visual Question Answering.
  • SNLI-VE: Visual Entailment.
  • Flickr30K & MSCOCO: Image-Text and Text-Image Retrieval.
  • NLVR2: Natural Language Visual Reasoning.

Key Insights and Ideas

  • Layer-wise Interaction Matters: Connecting intermediate layers facilitates "bottom-up" alignment. The cross-modal encoder can see how low-level visual features relate to syntactic structures before they are compressed into highly abstract representations.
  • Diversity of Attention: Through visualization (using KL divergence), the authors found that BridgeTower’s attention heads are significantly more diverse than those in models like METER (which only uses the last layer). This suggests that the bridge layers allow the model to attend to a wider variety of tokens and image patches simultaneously.
  • Efficiency: BridgeTower achieves superior results not by being "bigger" in terms of parameters, but by being "smarter" about how it uses existing pre-trained features.
  • Qualitative Advantage: In case studies (Figure 5), BridgeTower showed a better ability to focus on specific relevant details in an image (like a clock face) compared to models that only look at global, high-level features.
Visualization showing BridgeTower correctly attending to a clock to answer "What time is it?", whereas the baseline METER is distracted by the sky.

Key Results

  • New SOTA: With only 4M images for pre-training, BridgeTower-Base achieved 78.73% accuracy on VQAv2, outperforming the previous state-of-the-art model (METER) by 1.09% using the exact same data.
  • Scaling Success: When scaled up (BridgeTower-Large), it reached 81.15% on VQAv2, surpassing models pre-trained on orders-of-magnitude more data (e.g., models using 54M or even 1.8B images).
  • Retrieval Performance: On Flickr30K retrieval, it achieved an RSUM (sum of recall metrics) of 576.6, outperforming models like ALBEF (trained on 14M images).
  • Uni-modal Preservation: Interestingly, BridgeTower maintained better performance on purely visual (CIFAR) and purely textual (GLUE) benchmarks after VL pre-training compared to other fusion methods, suggesting the bridge layers interfere less with the original pre-trained knowledge.

Limitations and Future Work

  • Numerical Reasoning: The authors noted that BridgeTower still struggles with "Number" type questions in VQAv2 compared to models trained on much larger datasets.
  • Generative Tasks: This paper focused on discriminative tasks (choosing an answer or retrieving an image). The authors suggest that exploring bridge layers for generative tasks like image captioning is a promising next step.
  • Pre-training Objectives: The study stuck to standard MLM and ITM objectives for fair comparison; future work could incorporate newer objectives like Image-Text Contrastive (ITC) learning or Masked Image Modeling (MIM) to further boost performance.

Summary updated

Abstract

Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code and checkpoints are available at https://github.com/microsoft/BridgeTower.
BibTeX
@inproceedings{xu2023bridgetower,
  author = {Xiao Xu and Chenfei Wu and Shachar Rosenman and Vasudev Lal and Wanxiang Che and Nan Duan},
  title = {{BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26263},
}
Figure