Abstract
Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks.
Introduction
Current Vision-Language Pre-training (VLP) models face two primary bottlenecks that hinder their scalability and performance. First, they are largely dependent on high-quality, aligned image-caption pairs. Such datasets are expensive to curate and significantly smaller than the vast amounts of raw, unaligned images and text available on the web. Second, many models utilize region-based visual features extracted from pre-trained, frozen object detectors (e.g., Faster R-CNN). This two-stage approach is computationally intensive, restricted by the detector's fixed vocabulary, and prevents the visual encoder from being optimized end-to-end for multi-modal tasks.
UNIMO-2 addresses these challenges by introducing an end-to-end architecture that processes raw image patches and text tokens simultaneously. Crucially, it leverages "unpaired" data—images without captions and text without images—to build a more robust and generalized semantic understanding.
Grounded Learning Framework
The core innovation of UNIMO-2 is its Grounded Learning mechanism, designed to align visual and textual semantics across both paired and unpaired corpora.
Unified Architecture
The framework employs a ViT-based visual transformer and a RoBERTa-based text transformer. Unlike previous approaches that simply concatenate features, UNIMO-2 integrates these inputs into a Grounded Transformer.
Grounded Dictionary (GD)
At the heart of the model is a shared embedding space called the Grounded Dictionary—a matrix of latent vectors representing "grounded concepts." Using Vector Quantization (VQ), the model maps image patches and text tokens to the most relevant "grounded tokens" within this dictionary. These tokens serve as a common semantic language, bridging the gap between raw pixels and linguistic tokens.
Bottleneck Attention
Within the Grounded Transformer, cross-modal information exchange is restricted to these grounded tokens. This "bottleneck" forces the model to distill and exchange only the most significant shared semantics, enhancing robustness against noisy or weakly correlated image-text pairs.
Curriculum Training
To effectively initialize the grounded space, the authors utilize a curriculum training strategy. The model is first trained on aligned image-text pairs to establish a foundational grounded dictionary. Subsequently, it incorporates unaligned images via Visual Contrastive Learning and unaligned text through Masked Language Modeling.
Datasets and Evaluation
UNIMO-2 was pre-trained on a diverse set of data:
- Aligned Data: Approximately 4 million pairs from COCO, Visual Genome (VG), Conceptual Captions (CC), and SBU Captions.
- Unaligned Images: 15 million images from ImageNet-21k and OpenImages.
- Unaligned Text: 54GB of text from BookWiki and OpenWebText.
The model's performance was evaluated across a wide range of benchmarks, including cross-modal tasks (VQA v2.0, MS-COCO Captioning, SNLI-VE, Flickr30k Retrieval) and single-modal tasks (ImageNet-1k for vision, GLUE suite for NLP).
Key Results
UNIMO-2 demonstrates superior performance across both cross-modal and single-modal benchmarks.
Cross-Modal Excellence
As shown in the comparative results, UNIMO-2 achieves state-of-the-art performance on the VQA test-dev set (76.31) and COCO Captioning (131.2 CIDEr score). In zero-shot retrieval tasks, it significantly outperforms CLIP at similar data scales. For instance, on Zero-Shot Image Retrieval (ZS-IR), UNIMO-2 reached 72.7 R@1, compared to 65.1 for a baseline without the grounded dictionary.
Single-Modal Versatility
Unlike many VLP models that specialize solely in multi-modal tasks, UNIMO-2 remains highly competitive in single-modal domains. It achieved 66.3% zero-shot accuracy on ImageNet-1k and maintained performance levels comparable to RoBERTa-base on NLP tasks like MNLI.
Insights and Future Directions
The success of UNIMO-2 highlights several key insights:
- Semantic Bottlenecks: Forcing modality interaction through a discrete dictionary acts as a semantic filter, preventing overfitting to trivial correlations.
- Synergy of Unaligned Data: Learning from raw images and text acts as a force multiplier, improving cross-modal alignment by providing a broader context for grounded concepts.
- End-to-End Efficiency: Patch-based representations (ViT) are not only more scalable than region-based features but also more effective when paired with a strong alignment mechanism.
Future work includes scaling the model to even larger unaligned datasets and refining the grounded dictionary update mechanisms to capture increasingly fine-grained semantics. While efficient, the unified transformer architecture still requires significant computational resources for training at scale.
Summary updated