Low-resource Neural Machine Translation with Cross-modal Alignment

Zhe Yang, Qingkai Fang, Yang Feng

Abstract

Neural Machine Translation (NMT) typically requires millions of parallel sentence pairs to achieve high performance, a requirement that many of the world's languages cannot meet. This paper introduces a novel framework that utilizes images as a universal "pivot" to bridge the gap between high-resource and low-resource languages. By aligning multiple languages into a shared visual-textual space through coarse-to-fine contrastive learning, the researchers achieved significant improvements in zero-shot and few-shot translation quality without requiring massive parallel text corpora.

Introduction

Standard NMT models are inherently data-hungry. Existing workarounds, such as unsupervised NMT or multilingual NMT, still face high data bars or require complex auxiliary parallel data. This research identifies a missed opportunity in leveraging images. Because visual concepts are language-agnostic—a dog is a dog regardless of the language used to describe it—images can serve as a semantic bridge between languages that have never been seen together in a parallel text.

Conceptual overview of using images as a universal pivot for cross-modal alignment and zero-shot translation.

Methodology

The authors propose a cross-modal contrastive learning framework that integrates a Vision Transformer (ViT) as the image encoder and a standard Transformer for text processing. The core of the system is the Contrastive Module, which aligns representations through a two-stage process:

  1. Sentence-level (Coarse) Alignment: The model aligns the global representation of an image with the average representation of a sentence.
  2. Token-level (Fine) Alignment: Utilizing a "Selective Attention" mechanism, the model learns to correlate specific image patches with individual words (e.g., matching a visual dog to the word "chien"). This fine-grained alignment is critical for maintaining translation accuracy.
Detailed architecture of the cross-modal NMT model, highlighting the Contrastive Module and Selective Attention mechanism.

The Pivot Strategy

The model is trained on high-resource parallel data (e.g., German-English) and image-text pairs for the target low-resource language (e.g., French-Image). Because both German and French text are aligned to the same visual concepts, the model effectively "learns" to translate French to English via the visual bridge.

Experimental Setup

The framework was evaluated using several benchmarks:

  • Multi30K: A standard multi-modal NMT dataset with captions in English, German, French, and Czech.
  • MsCOCO & VizWiz: Large-scale image captioning datasets.
  • Pseudo Data: To scale training, the authors used high-quality NMT models to translate English captions from MsCOCO and VizWiz into the target low-resource languages.

Key Results and Insights

The proposed S+T-CTR model demonstrated dramatic improvements over text-only baselines:

  • Zero-Shot Translation: On the FR→EN task, where the model had never seen French-English pairs, it improved from a baseline of 0.30 BLEU to 17.76 BLEU.
  • Few-Shot Translation: When provided with a minimal amount of parallel data (e.g., 100 sentences), the model consistently outperformed baselines across all tested languages.
  • Semantic Grounding: Visualization confirmed that the model successfully collapsed different language representations for the same concept into the same area of the embedding space. Furthermore, including the target language in the contrastive learning process was found to significantly boost mapping accuracy.

Limitations and Future Work

While effective, the method relies on the quality of pseudo-data generated for training. Additionally, visual grounding is most effective for descriptive language; abstract concepts (e.g., "justice") remain difficult to capture through images alone. Future work may explore applying this cross-modal transfer to other tasks like cross-lingual information retrieval.

Summary updated

Abstract

How to achieve neural machine translation with limited parallel data? Existing techniques often rely on large-scale monolingual corpora, which is impractical for some low-resource languages. In this paper, we turn to connect several low-resource languages to a particular high-resource one by additional visual modality. Specifically, we propose a cross-modal contrastive learning method to learn a shared space for all languages, where both a coarse-grained sentence-level objective and a fine-grained token-level one are introduced. Experimental results and further analysis show that our method can effectively learn the cross-modal and cross-lingual alignment with a small amount of image-text pairs and achieves significant improvements over the text-only baseline under both zero-shot and few-shot scenarios.
BibTeX
@inproceedings{yang2022low,
  author = {Zhe Yang and Qingkai Fang and Yang Feng},
  title = {{Low-resource Neural Machine Translation with Cross-modal Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=RRm5MFZFW_},
}
Figure