Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

Yan Zeng, Wangchunshu Zhou, Ao Luo, Xinsong Zhang

Abstract

In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is motivated by a key observation that cross-lingual and cross-modal pre-training share the same goal of aligning two different views of the same object into a common semantic space. To this end, the cross-view language modeling framework considers both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as two different views of the same object, and trains the model to align the two views by maximizing the mutual information between them with conditional masked language modeling and contrastive learning. We pre-train CCLM, a Cross-lingual Cross-modal Language Model, with the cross-view language modeling framework. Empirical results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual image-text retrieval datasets show that while conceptually simpler, CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. Moreover, CCLM is the first multi-lingual multi-modal pre-trained model that surpasses the translate-test performance of representative English vision-language models by zero-shot cross-lingual transfer.

Introduction

The development of multi-lingual multi-modal AI aims to create systems capable of understanding images and text across diverse languages. A significant hurdle is the scarcity of high-quality image-caption pairs for most languages outside of English. Previous state-of-the-art models, such as M3P and UC2, attempted to bridge this gap by "pivoting" on English or images. However, these methods often failed to outperform a simple "translate-test" baseline—where non-English queries are translated to English before being processed by an English-only model. This performance gap suggested that existing models were not effectively learning a truly shared semantic space across languages and modalities.

The Cross-View Framework

The proposed Cross-View Language Modeling (CVLM) framework is built on the intuition that image-caption pairs and parallel sentence pairs are essentially different views of the same semantic concept.

Architecture and Objectives

CCLM utilizes a modular Transformer-based architecture consisting of a Vision Transformer (ViT) image encoder, a multi-lingual XLM-R text encoder, and a shared Fusion Transformer. The fusion module employs cross-attention to align the two views.

The CCLM architecture unifies (a) image-caption pairs and (b) parallel text pairs under a single training framework with shared objectives.

The model is trained using three unified objectives applied to both cross-modal and cross-lingual data:

  • Contrastive Loss ($L_{cl}$): Aligns global representations in a shared semantic space.
  • Matching Loss ($L_{match}$): A binary classification task to determine if a pair of views is correctly matched.
  • Conditional Masked Language Modeling ($L_{mlm}$): Encourages the model to predict masked tokens in one view using context from the other.

Experimental Evaluation

CCLM was pre-trained on a combination of multi-modal data (CC3M and X2-VLM) and multi-lingual data (19 million parallel sentence pairs from WikiMatrix). Evaluation was conducted on the IGLUE benchmark, which spans tasks like visual entailment (XVNLI), grounded question answering (xGQA), and visual reasoning (MaRVL).

Performance Gains

CCLM sets a new state-of-the-art across the IGLUE benchmark, achieving an average absolute improvement of over 10% compared to previous models like UC2. Notably, CCLM is the first model to consistently beat the "translate-test" performance of representative English vision-language models via zero-shot cross-lingual transfer.

Radar charts comparing CCLM (Ours 3M) with UC2 across (a) various IGLUE tasks and (b) different languages, demonstrating broad performance improvements.

Key results include:

  • Retrieval: On the xFlickr&CO Image Retrieval task, CCLM (base) achieved a score of 76.94, significantly outperforming UC2's 20.31.
  • Scaling: The framework scales effectively; the CCLM-large variant (970M parameters) showed continued performance gains with more data.

Conclusion

CCLM demonstrates that treating parallel text as a "first-class citizen" alongside image-text pairs is a highly effective strategy for multi-lingual vision-language pre-training. By unifying these tasks under a cross-view framework, the model successfully shrinks the "transfer gap" between English and other languages. Future work may extend this unified approach to other modalities like video and audio.

Summary updated

Abstract

In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is motivated by a key observation that cross-lingual and cross-modal pre-training share the same goal of aligning two different views of the same object into a common semantic space. To this end, the cross-view language modeling framework considers both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as two different views of the same object, and trains the model to align the two views by maximizing the mutual information between them with conditional masked language modeling and contrastive learning. We pre-train CCLM, a Cross-lingual Cross-modal Language Model, with the cross-view language modeling framework. Empirical results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual image-text retrieval datasets show that while conceptually simpler, CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. Moreover, CCLM is the first multi-lingual multi-modal pre-trained model that surpasses the translate-test performance of representative English vision-language models by zero-shot cross-lingual transfer.
BibTeX
@inproceedings{zeng2023cross,
  author = {Yan Zeng and Wangchunshu Zhou and Ao Luo and Xinsong Zhang},
  title = {{Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=Y2nYqTspO8},
}
Figure