Abstract
Achieving high-quality neural machine translation (NMT) typically requires massive parallel corpora, a resource that is unavailable for the vast majority of the world's languages. While existing unsupervised techniques attempt to bridge this gap using large-scale monolingual data, such data is also often scarce for truly low-resource languages. This paper proposes a novel framework that leverages the visual modality as a "universal pivot" to connect low-resource languages to high-resource ones. By employing a two-stage cross-modal contrastive learning method—incorporating both coarse-grained sentence-level and fine-grained token-level objectives—we demonstrate that a shared embedding space can be learned across multiple languages. Our experimental results show that this approach enables functional zero-shot translation and provides significant performance gains in few-shot scenarios, outperforming traditional text-only baselines.
Introduction: The Data Hunger of NMT
The "data hunger" of Neural Machine Translation remains a primary barrier to global digital inclusivity. Standard NMT models generally require millions of parallel sentence pairs to achieve fluency. For most of the thousands of languages spoken globally, such datasets do not exist. Even unsupervised NMT, which relies on monolingual text, often requires over a million sentences to be effective—a threshold still too high for many dialects.
This research identifies a unique opportunity: while parallel text is scarce, image-text pairs (captions) are increasingly abundant. Because visual concepts—such as a "dog running in the snow"—are universal regardless of the language used to describe them, images can serve as a semantic bridge. We explore whether this visual grounding can connect a low-resource language to a high-resource one, effectively bypassing the need for massive bilingual text corpora.
Methodology: Cross-Modal Contrastive Learning
We propose a framework centered on cross-modal contrastive learning, designed to align different languages into a single, shared vector space using images as the anchor. This process occurs in two distinct stages:
1. Sentence-level Alignment (Coarse-grained)
In this stage, the model learns to map an entire sentence and its corresponding image to similar locations in the shared space. Using a contrastive "matching" logic, the representation of a caption (e.g., in French) is optimized to be close to its corresponding image while remaining distant from unrelated images. This establishes a general semantic mapping between the visual and linguistic domains.
2. Token-level Alignment (Fine-grained)
To ensure the model captures specific nuances and avoids omitting key entities during translation, we introduced a Selective Attention module. This allows the model to correlate specific words (e.g., "dog") with specific patches or regions of an image. This fine-grained alignment prevents the model from generating grammatically plausible but semantically incomplete translations.
System Architecture
Our architecture utilizes a Vision Transformer (ViT) pre-trained via CLIP as the image encoder, while the text encoders and decoders utilize standard Transformer architectures. The core strategy involves training the model on a high-resource language (such as German) with parallel English text, while simultaneously using image-text pairs to align low-resource languages (such as French or Czech) into the same shared space.
Experimental Setup and Datasets
We evaluated the framework using three primary multi-modal datasets:
- Multi30K: A benchmark containing images with captions in English, German, French, and Czech.
- MsCOCO & VizWiz: Large-scale English image-captioning datasets.
- Pseudo Data Generation: To overcome the scarcity of non-English captions, we utilized existing MT models to translate English captions from MsCOCO and VizWiz into the target low-resource languages, creating a larger synthetic training pool.
Key Insights and Results
Our findings highlight the efficacy of visual pivoting in data-starved environments:
- Zero-Shot Breakthroughs: In the French-to-English (FR→EN) task, a standard text-only baseline failed completely, yielding a BLEU score of 0.21. Our full cross-modal model achieved 15.76 BLEU, demonstrating that functional translation is possible without any initial parallel text.
- Few-Shot Efficiency: When provided with a mere 100 parallel sentences, our model's performance improved significantly, outperforming the baseline by nearly 10 BLEU points. This suggests that visual alignment provides a superior starting point for fine-tuning.
- The Importance of the Target Language: A critical insight was that including the target language (English) in the contrastive learning loop—aligning English text with images alongside the source languages—significantly boosted performance. This helps "close the loop" between the source and target embedding spaces.
- Visual Grounding: Qualitative analysis of attention maps confirmed that the model correctly associates specific words with visual regions (e.g., the word "maison" correctly triggers attention on a house in the image), proving that the cross-modal alignment is semantically accurate.
Limitations and Future Work
While promising, the approach faces several challenges. The reliance on pseudo-data (machine-translated captions) means that errors in the initial translation can propagate through the model. Furthermore, while the model excels at capturing semantics, it still struggles with perfect grammatical syntax in pure zero-shot settings; we observed that even a small amount of real parallel data is necessary to refine the model's grammatical output.
Future research will focus on applying this visual pivoting strategy to a broader variety of non-European language pairs and exploring its utility in more complex multi-modal reasoning tasks.
Summary updated