Abstract
Current vision-and-language (V&L) models often rely on task-specific architectures, adding custom output "heads" for different objectives like classification or region scoring. We present VL-T5 and VL-BART, a unified framework that treats all V&L tasks as a text generation problem. By utilizing a single language modeling objective, these models can perform diverse tasks—from answering questions to grounding objects—within a single architecture. Our approach demonstrates superior generalization to rare, open-ended scenarios and achieves competitive performance with significantly reduced architectural complexity.
Introduction: The Fragmentation of V&L Architectures
The prevailing paradigm in vision-and-language research involves "Frankenstein" architectures. To perform Visual Question Answering (VQA), researchers typically append a multi-label classifier to a transformer. For Visual Grounding, a region-scoring head is required, while image captioning necessitates a full language decoder. This fragmentation introduces several challenges:
- Architectural Complexity: Each new task requires a bespoke output layer.
- Limited Generalization: Discriminative models are often restricted to a fixed vocabulary of frequent answers (e.g., the top 3,000), failing on "long-tail" or rare responses.
- Multi-tasking Friction: Training a single model on heterogeneous tasks is difficult when each requires different parameters and loss functions.
To address these issues, we propose a unified "text-to-text" format where all tasks are framed as generating a sequence of tokens.
Methods: A Unified Generative Framework
We extend the pre-trained T5 and BART language models into the multimodal domain. The core of the framework is a standard Transformer encoder-decoder architecture.
Multimodal Input Representation
- Visual Input: Instead of raw pixels, the model processes 36 object regions extracted via a Faster R-CNN. These regions are converted into embeddings comprising visual features, spatial coordinates, and image IDs.
- Text Input: The model receives a text prompt with a task-specific prefix, such as
"vqa: what is the man jumping over?"or"visual grounding: yellow fire hydrant".
Visual Sentinel Tokens
A key innovation is the introduction of visual sentinel tokens (e.g., <vis_1>, <vis_2>) into the model's vocabulary. These tokens allow the model to "point" to specific image regions by generating the corresponding text string. This unifies "speaking" (text generation) and "pointing" (grounding) into a single mechanism.
Experimental Setup and Results
The models were pre-trained on an aggregate of 9.18 million image-text pairs from sources including MS COCO, Visual Genome, VQA v2.0, and GQA. Evaluation was conducted across seven downstream benchmarks, including NLVR², VCR, and Multi30K.
Key Findings
- Generalization to Rare Answers: In VQA splits containing rare answers, the generative VL-T5 significantly outperformed discriminative baselines (13.1% vs. 7.1% accuracy), highlighting the advantage of open-ended generation over fixed-set classification.
- Competitive Performance: Despite its unified nature, VL-T5 achieved results comparable to state-of-the-art specialized models like UNITER and Oscar. For instance, on VCR (QA→R), it reached 58.9% accuracy.
- Effective Multi-Tasking: A single VL-T5 model trained on all seven tasks simultaneously performed nearly as well as seven individually optimized models, demonstrating that reasoning skills are shared across tasks.
Discussion and Future Work
While the framework simplifies V&L modeling, it currently relies on frozen visual features from a Faster R-CNN, creating a bottleneck. Future iterations could explore end-to-end training directly from pixels. Additionally, more sophisticated prompt tuning may further enhance performance. Interestingly, VL-BART showed weaker grounding capabilities than VL-T5, likely due to differences in positional embedding strategies.
In conclusion, this work represents a significant step toward general-purpose visual assistants capable of reading, seeing, and reasoning within a single, unified brain.
Summary updated