Transformer-Exclusive Cross-Modal Representation for Vision and Language

Andrew Shin, Takuya NarihiraACL 2021 · OpenReview

TL;DR

This paper explores a novel approach to cross-modal learning for vision and language tasks by exclusively using Transformer architectures, eliminating the need for traditional Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The authors demonstrate that this "transformer-exclusive" approach can achieve comparable performance to existing methods, paving the way for more unified and potentially efficient multimodal models.

What Problem Does This Paper Address?

Traditionally, learning representations that bridge vision and language has relied on a combination of CNNs for image processing and RNNs for text processing. While this CNN-RNN paradigm has been successful, the advent of the Transformer architecture has revolutionized Natural Language Processing (NLP) and shown promise in vision tasks. This paper investigates whether Transformers alone can handle cross-modal vision-language tasks, aiming to simplify the architecture and potentially improve efficiency by removing the need for specialized CNN and RNN components.

Methods

The core of the proposed method involves using separate Transformer encoders for both image and language modalities. For images, the input is divided into patches, which are then projected into embeddings and fed into a Transformer encoder, similar to the Vision Transformer (ViT) approach. For language, a standard off-the-shelf BERT model is employed. Crucially, the paper emphasizes a simple concatenation of the final representations from these two Transformer encoders as the "common embedding scheme," deliberately avoiding complex fusion mechanisms to focus on the inherent capabilities of the Transformer-exclusive representations.

Datasets

The paper evaluates its model on several standard cross-modal visuolinguistic tasks:

  • Visual Question Answering (VQA)
  • Visual Commonsense Reasoning (VCR)
  • Natural Language for Visual Reasoning (NLVR2)

The models are trained and tested using datasets associated with these tasks, such as ImageNet for pre-training the vision component.

Key Insights and Ideas

  • Architectural Unification: The paper champions the idea of using a single architectural paradigm (Transformers) for both vision and language. This could lead to more streamlined model development and potentially better transfer learning across modalities.
  • Transformer for Vision is Sufficient: Building on prior work like ViT, the paper reinforces that Transformers are capable of handling image processing tasks effectively, even without convolutions.
  • Simple Fusion Works: The authors demonstrate that a straightforward concatenation of Transformer-generated features from vision and language can yield competitive results, suggesting that the power lies in the individual modality representations rather than complex fusion strategies.
  • Efficiency Potential: By removing CNNs and RNNs, the proposed approach has the potential for computational and data efficiency, especially when considering the computational cost of region detectors often used in prior methods.

Key Results

The paper reports comparable performance to state-of-the-art cross-modal models on VQA, VCR, and NLVR2 tasks. While not explicitly claiming state-of-the-art, the results are noteworthy because the proposed model achieves this without extensive cross-modal pre-training or complex embedding schemes, which are common in competing methods. For instance, the model performs well despite not having seen the large amounts of image-caption pairs used for pre-training in other models.

Limitations and Future Work

The authors acknowledge that their model's performance falls below that of some state-of-the-art models. They attribute this, in part, to the deliberate choice of using the smallest baseline models for each component and a simple concatenation scheme. They suggest that significant performance boosts could be achieved by:

  • Using larger pre-trained Transformer models.
  • Employing more sophisticated common embedding schemes.
  • Leveraging more extensive cross-modal pre-training data.

They also propose that the architectural integrity of using Transformers for both modalities is a promising direction for future research, encouraging the development of more advanced models with novel ideas.

Summary updated

BibTeX
@inproceedings{shin2021transformer,
  author = {Andrew Shin and Takuya Narihira},
  title = {{Transformer-Exclusive Cross-Modal Representation for Vision and Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=hoItvDBmyB},
}
Figure