UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, Haifeng Wang

Abstract

Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks.

Introduction

Current Vision-Language Pre-training (VLP) models face two primary bottlenecks that hinder their scalability and performance. First, they are largely dependent on high-quality, aligned image-caption pairs. Such datasets are expensive to curate and significantly smaller than the vast amounts of raw, unaligned images and text available on the web. Second, many models utilize region-based visual features extracted from pre-trained, frozen object detectors (e.g., Faster R-CNN). This two-stage approach is computationally intensive, restricted by the detector's fixed vocabulary, and prevents the visual encoder from being optimized end-to-end for multi-modal tasks.

UNIMO-2 addresses these challenges by introducing an end-to-end architecture that processes raw image patches and text tokens simultaneously. Crucially, it leverages "unpaired" data—images without captions and text without images—to build a more robust and generalized semantic understanding.

Grounded Learning Framework

The core innovation of UNIMO-2 is its Grounded Learning mechanism, designed to align visual and textual semantics across both paired and unpaired corpora.

UNIMO-2 architecture for grounded learning on both paired and unpaired data.

Unified Architecture

The framework employs a ViT-based visual transformer and a RoBERTa-based text transformer. Unlike previous approaches that simply concatenate features, UNIMO-2 integrates these inputs into a Grounded Transformer.

Grounded Dictionary (GD)

At the heart of the model is a shared embedding space called the Grounded Dictionary—a matrix of latent vectors representing "grounded concepts." Using Vector Quantization (VQ), the model maps image patches and text tokens to the most relevant "grounded tokens" within this dictionary. These tokens serve as a common semantic language, bridging the gap between raw pixels and linguistic tokens.

Bottleneck Attention

Within the Grounded Transformer, cross-modal information exchange is restricted to these grounded tokens. This "bottleneck" forces the model to distill and exchange only the most significant shared semantics, enhancing robustness against noisy or weakly correlated image-text pairs.

Curriculum Training

To effectively initialize the grounded space, the authors utilize a curriculum training strategy. The model is first trained on aligned image-text pairs to establish a foundational grounded dictionary. Subsequently, it incorporates unaligned images via Visual Contrastive Learning and unaligned text through Masked Language Modeling.

Datasets and Evaluation

UNIMO-2 was pre-trained on a diverse set of data:

  • Aligned Data: Approximately 4 million pairs from COCO, Visual Genome (VG), Conceptual Captions (CC), and SBU Captions.
  • Unaligned Images: 15 million images from ImageNet-21k and OpenImages.
  • Unaligned Text: 54GB of text from BookWiki and OpenWebText.

The model's performance was evaluated across a wide range of benchmarks, including cross-modal tasks (VQA v2.0, MS-COCO Captioning, SNLI-VE, Flickr30k Retrieval) and single-modal tasks (ImageNet-1k for vision, GLUE suite for NLP).

Key Results

UNIMO-2 demonstrates superior performance across both cross-modal and single-modal benchmarks.

Performance comparison on various cross-modal downstream tasks.

Cross-Modal Excellence

As shown in the comparative results, UNIMO-2 achieves state-of-the-art performance on the VQA test-dev set (76.31) and COCO Captioning (131.2 CIDEr score). In zero-shot retrieval tasks, it significantly outperforms CLIP at similar data scales. For instance, on Zero-Shot Image Retrieval (ZS-IR), UNIMO-2 reached 72.7 R@1, compared to 65.1 for a baseline without the grounded dictionary.

Single-Modal Versatility

Unlike many VLP models that specialize solely in multi-modal tasks, UNIMO-2 remains highly competitive in single-modal domains. It achieved 66.3% zero-shot accuracy on ImageNet-1k and maintained performance levels comparable to RoBERTa-base on NLP tasks like MNLI.

Insights and Future Directions

The success of UNIMO-2 highlights several key insights:

  • Semantic Bottlenecks: Forcing modality interaction through a discrete dictionary acts as a semantic filter, preventing overfitting to trivial correlations.
  • Synergy of Unaligned Data: Learning from raw images and text acts as a force multiplier, improving cross-modal alignment by providing a broader context for grounded concepts.
  • End-to-End Efficiency: Patch-based representations (ViT) are not only more scalable than region-based features but also more effective when paired with a strong alignment mechanism.

Future work includes scaling the model to even larger unaligned datasets and refining the grounded dictionary update mechanisms to capture increasingly fine-grained semantics. While efficient, the unified transformer architecture still requires significant computational resources for training at scale.

Summary updated

Abstract

Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks. Our code and models are public at the UNIMO project page https://unimo-ptm.github.io/.
BibTeX
@inproceedings{li2022unimo,
  author = {Wei Li and Can Gao and Guocheng Niu and Xinyan Xiao and Hao Liu and Jiachen Liu and Hua Wu and Haifeng Wang},
  title = {{UNIMO-2: End-to-End Unified Vision-Language Grounded Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=DNjjnSyRo7},
}
Figure