Unifying Vision-and-Language Tasks via Text Generation

Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal

Abstract

Current vision-and-language (V&L) models often rely on task-specific architectures, adding custom output "heads" for different objectives like classification or region scoring. We present VL-T5 and VL-BART, a unified framework that treats all V&L tasks as a text generation problem. By utilizing a single language modeling objective, these models can perform diverse tasks—from answering questions to grounding objects—within a single architecture. Our approach demonstrates superior generalization to rare, open-ended scenarios and achieves competitive performance with significantly reduced architectural complexity.

Introduction: The Fragmentation of V&L Architectures

The prevailing paradigm in vision-and-language research involves "Frankenstein" architectures. To perform Visual Question Answering (VQA), researchers typically append a multi-label classifier to a transformer. For Visual Grounding, a region-scoring head is required, while image captioning necessitates a full language decoder. This fragmentation introduces several challenges:

  1. Architectural Complexity: Each new task requires a bespoke output layer.
  2. Limited Generalization: Discriminative models are often restricted to a fixed vocabulary of frequent answers (e.g., the top 3,000), failing on "long-tail" or rare responses.
  3. Multi-tasking Friction: Training a single model on heterogeneous tasks is difficult when each requires different parameters and loss functions.
Comparison of traditional task-specific heads versus the proposed unified language modeling head.

To address these issues, we propose a unified "text-to-text" format where all tasks are framed as generating a sequence of tokens.

Methods: A Unified Generative Framework

We extend the pre-trained T5 and BART language models into the multimodal domain. The core of the framework is a standard Transformer encoder-decoder architecture.

Multimodal Input Representation

  • Visual Input: Instead of raw pixels, the model processes 36 object regions extracted via a Faster R-CNN. These regions are converted into embeddings comprising visual features, spatial coordinates, and image IDs.
  • Text Input: The model receives a text prompt with a task-specific prefix, such as "vqa: what is the man jumping over?" or "visual grounding: yellow fire hydrant".
The VL-T5/VL-BART architecture and visual embedding process.

Visual Sentinel Tokens

A key innovation is the introduction of visual sentinel tokens (e.g., <vis_1>, <vis_2>) into the model's vocabulary. These tokens allow the model to "point" to specific image regions by generating the corresponding text string. This unifies "speaking" (text generation) and "pointing" (grounding) into a single mechanism.

Experimental Setup and Results

The models were pre-trained on an aggregate of 9.18 million image-text pairs from sources including MS COCO, Visual Genome, VQA v2.0, and GQA. Evaluation was conducted across seven downstream benchmarks, including NLVR², VCR, and Multi30K.

Key Findings

  • Generalization to Rare Answers: In VQA splits containing rare answers, the generative VL-T5 significantly outperformed discriminative baselines (13.1% vs. 7.1% accuracy), highlighting the advantage of open-ended generation over fixed-set classification.
  • Competitive Performance: Despite its unified nature, VL-T5 achieved results comparable to state-of-the-art specialized models like UNITER and Oscar. For instance, on VCR (QA→R), it reached 58.9% accuracy.
  • Effective Multi-Tasking: A single VL-T5 model trained on all seven tasks simultaneously performed nearly as well as seven individually optimized models, demonstrating that reasoning skills are shared across tasks.

Discussion and Future Work

While the framework simplifies V&L modeling, it currently relies on frozen visual features from a Faster R-CNN, creating a bottleneck. Future iterations could explore end-to-end training directly from pixels. Additionally, more sophisticated prompt tuning may further enhance performance. Interestingly, VL-BART showed weaker grounding capabilities than VL-T5, likely due to differences in positional embedding strategies.

In conclusion, this work represents a significant step toward general-purpose visual assistants capable of reading, seeing, and reasoning within a single, unified brain.

Summary updated

Abstract

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual quest...
BibTeX
@inproceedings{cho2021unifying,
  author = {Jaemin Cho and Jie Lei and Hao Tan and Mohit Bansal},
  title = {{Unifying Vision-and-Language Tasks via Text Generation}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=UuRd7lWbv_},
}
Figure