UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, Haifeng Wang

Abstract

UNIMO introduces a unified-modal pre-training architecture designed to bridge the gap between single-modal (text-only or image-only) and multi-modal (image-text pairs) learning. By leveraging Cross-Modal Contrastive Learning (CMCL) on a massive mix of paired and unpaired data, UNIMO creates a shared semantic space where visual and textual knowledge mutually enhance each other. This approach achieves state-of-the-art performance across both pure natural language processing (NLP) tasks and vision-language tasks, demonstrating that cross-modal pre-training can benefit single-modal understanding and generation.

Introduction

Historically, AI models have been siloed by modality. Computer Vision (CV) models like ResNet are typically pre-trained on massive image datasets, while Natural Language Processing (NLP) models like BERT and RoBERTa utilize vast text corpora. Recently, multi-modal models such as ViLBERT and UNITER have emerged to tackle vision-language tasks by training on image-text pairs. However, these models often rely exclusively on limited, expensive paired datasets and can overfit to the multi-modal format, leading to significant performance drops when applied to pure language tasks like text summarization.

UNIMO addresses this gap by proposing a single, unified AI system capable of effectively processing unpaired single-modal data alongside paired multi-modal data. The goal is to allow different modalities to enrich one another rather than being constrained by the availability of paired data.

Methodology

UNIMO utilizes a multi-layer self-attention Transformer to process text tokens, image regions (extracted via Faster R-CNN), or a combination of both. The training process integrates three primary objectives:

  1. Visual Learning: The model masks highly overlapping image regions and is tasked with reconstructing their visual features and predicting their object classes.
  2. Language Learning: This involves bidirectional masked prediction (masking complete words or phrases) and auto-regressive sequence-to-sequence (Seq2Seq) generation.
  3. Cross-Modal Contrastive Learning (CMCL): As the core novelty, CMCL aligns visual and textual representations into a shared semantic space.
The UNIMO architecture processes image collections, text corpora, and image-text pairs through a Unified-Modal Transformer to create a shared semantic space.

To enhance CMCL, the authors introduce two key data augmentation techniques:

  • Text Rewriting: To move beyond simple random negatives, UNIMO parses captions into scene graphs and swaps specific objects, attributes, or relations to create "hard" negative examples. Back-translation is also used to generate diverse positive examples.
  • Image/Text Retrieval: To incorporate single-modal data, UNIMO retrieves semantically related unpaired images and texts to augment existing image-text pairs, providing richer background context.
Cross-Modal Contrastive Learning (CMCL) uses text rewriting and retrieval to generate positive and negative pairs for fine-grained alignment.

Key Insights

A significant finding of this work is "modality cross-pollination." Training on unpaired images improves performance on pure text tasks, and vice versa. Visual knowledge helps ground language; for instance, the model learns semantic proximity between words like "foot" and "soccer" through visual context. Furthermore, the use of hard negatives forces the model to achieve fine-grained alignment, requiring it to scrutinize image details rather than relying on coarse background clues.

Experiments and Results

UNIMO was pre-trained on a massive dataset including BookWiki and OpenWebText (text), OpenImages and COCO (images), and approximately 9 million image-text pairs (COCO, Visual Genome, Conceptual Captions, SBU Captions).

The model demonstrates superior performance across diverse benchmarks:

  • Multi-Modal Tasks: UNIMO-large achieved state-of-the-art results on Flickr30k, outperforming ERNIE-ViL-large by +1.34 R@1 on image retrieval. It also surpassed the Oscar model by over 2 BLEU4 points on COCO Image Captioning.
  • Single-Modal Tasks: Unlike previous multi-modal architectures, UNIMO remains highly competitive in pure NLP. It matches or beats models like RoBERTa and UniLM, scoring 84.9 on CoQA and 96.8 on SST-2.

Ablation studies confirmed that removing unpaired text hindered multi-modal performance, while removing unpaired images degraded pure NLP performance, proving that modalities mutually enhance each other during pre-training.

Conclusion

While UNIMO represents a significant step toward unified-modal learning, it still relies on an external object detector (Faster R-CNN) for visual feature extraction, which prevents end-to-end learning from raw pixels. Future work will focus on bypassing this bottleneck and scaling the model and data further to achieve even deeper integration of visual and linguistic knowledge.

Summary updated

BibTeX
@inproceedings{li2021unimo,
  author = {Wei Li and Can Gao and Guocheng Niu and Xinyan Xiao and Hao Liu and Jiachen Liu and Hua Wu and Haifeng Wang},
  title = {{UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.202},
}
Figure