Abstract
Neural Machine Translation (NMT) typically requires millions of parallel sentence pairs to achieve high performance, a requirement that many of the world's languages cannot meet. This paper introduces a novel framework that utilizes images as a universal "pivot" to bridge the gap between high-resource and low-resource languages. By aligning multiple languages into a shared visual-textual space through coarse-to-fine contrastive learning, the researchers achieved significant improvements in zero-shot and few-shot translation quality without requiring massive parallel text corpora.
Introduction
Standard NMT models are inherently data-hungry. Existing workarounds, such as unsupervised NMT or multilingual NMT, still face high data bars or require complex auxiliary parallel data. This research identifies a missed opportunity in leveraging images. Because visual concepts are language-agnostic—a dog is a dog regardless of the language used to describe it—images can serve as a semantic bridge between languages that have never been seen together in a parallel text.
Methodology
The authors propose a cross-modal contrastive learning framework that integrates a Vision Transformer (ViT) as the image encoder and a standard Transformer for text processing. The core of the system is the Contrastive Module, which aligns representations through a two-stage process:
- Sentence-level (Coarse) Alignment: The model aligns the global representation of an image with the average representation of a sentence.
- Token-level (Fine) Alignment: Utilizing a "Selective Attention" mechanism, the model learns to correlate specific image patches with individual words (e.g., matching a visual dog to the word "chien"). This fine-grained alignment is critical for maintaining translation accuracy.
The Pivot Strategy
The model is trained on high-resource parallel data (e.g., German-English) and image-text pairs for the target low-resource language (e.g., French-Image). Because both German and French text are aligned to the same visual concepts, the model effectively "learns" to translate French to English via the visual bridge.
Experimental Setup
The framework was evaluated using several benchmarks:
- Multi30K: A standard multi-modal NMT dataset with captions in English, German, French, and Czech.
- MsCOCO & VizWiz: Large-scale image captioning datasets.
- Pseudo Data: To scale training, the authors used high-quality NMT models to translate English captions from MsCOCO and VizWiz into the target low-resource languages.
Key Results and Insights
The proposed S+T-CTR model demonstrated dramatic improvements over text-only baselines:
- Zero-Shot Translation: On the FR→EN task, where the model had never seen French-English pairs, it improved from a baseline of 0.30 BLEU to 17.76 BLEU.
- Few-Shot Translation: When provided with a minimal amount of parallel data (e.g., 100 sentences), the model consistently outperformed baselines across all tested languages.
- Semantic Grounding: Visualization confirmed that the model successfully collapsed different language representations for the same concept into the same area of the embedding space. Furthermore, including the target language in the contrastive learning process was found to significantly boost mapping accuracy.
Limitations and Future Work
While effective, the method relies on the quality of pseudo-data generated for training. Additionally, visual grounding is most effective for descriptive language; abstract concepts (e.g., "justice") remain difficult to capture through images alone. Future work may explore applying this cross-modal transfer to other tasks like cross-lingual information retrieval.
Summary updated