From Text to Multi-Modal: Advancing Low-Resource-Language Translation through Synthetic Data Generation and Cross-Modal Alignments

Bushi Xiao, Qian Shen, Daisy Zhe WangNAACL 2025

TL;DR

This paper tackles the challenge of low-resource language translation by introducing a novel method to automatically generate multimodal datasets. By leveraging text-to-image models and LLMs, they create visually-grounded parallel corpora, which then power a new multimodal translation model that significantly outperforms text-only baselines.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has seen tremendous progress, largely driven by massive datasets and powerful pre-trained models. However, this progress has not trickled down to low-resource languages, especially those in Africa. Existing multimodal NMT research for these languages is hampered by a scarcity of aligned multimodal datasets, which are expensive and time-consuming to create manually. Furthermore, even when multimodal data exists, ensuring semantic consistency between text and images remains a significant hurdle, leading to translation errors. This paper aims to bridge this gap by creating a scalable, cost-effective way to generate multimodal datasets for low-resource languages and developing a model that effectively utilizes this visual information.

Methods

The authors propose a two-pronged approach:

  1. Synthetic Multimodal Dataset Generation: Instead of relying on existing image-text pairs, they use state-of-the-art text-to-image models (Stable Diffusion 3.5 Large Turbo) to generate visuals. The process starts with existing text-only parallel corpora for low-resource languages and English. An LLM (Llama 3) is used to generate detailed "scene descriptions" from the English sentences. These descriptions then serve as prompts for the text-to-image model, creating aligned visual content. This generative pipeline is designed to ensure better semantic consistency, even for abstract concepts.
  2. Multimodal Translation Model: They introduce a new multimodal translation framework built upon the NLLB (No Language Left Behind) model. This architecture uses a dual-path approach:
  3. Textual Path: A fine-tuned NLLB model handles the low-resource language to English translation.
  4. Visual Path: A pre-trained BLIP model extracts features from the generated images and produces English captions.
  5. LLM-based Quality Supervision: A few-shot LLM (Llama 3) acts as a quality supervisor. It evaluates the semantic alignment between the NLLB translation and the image context (via BLIP captions). If the alignment score falls below a threshold, the LLM is prompted to regenerate the translation, aiming to improve semantic accuracy and visual grounding.

Datasets

The paper utilizes text-only parallel translation datasets for four African low-resource languages: Hausa, Kanuri, Tigrinya, and Yorùbá, paired with English. These datasets were sourced from Tatoeba and the AI4D Yorùbá Machine Translation Challenge on Zindi. For Kanuri and Tigrinya, they used 5,000 samples each, and for Hausa and Yorùbá, approximately 10,000 samples each. 25% of these were set aside for testing, with the rest used for training. The core contribution here is the creation of a new multimodal dataset derived from these text-only sources using their proposed generative pipeline.

Key Insights and Ideas

  • Visual Grounding for Low-Resource Languages: The core idea is that visual information can significantly aid translation, especially when text-based resources are scarce. By generating synthetic visual data, they overcome the bottleneck of acquiring real-world aligned data.
  • LLMs as Creative Data Augmenters and Quality Controllers: LLMs are not just used for translation but also for generating descriptive prompts for image synthesis and for supervising the translation quality by checking semantic alignment with visuals. This dual role highlights the versatility of LLMs.
  • Addressing Abstract Concepts: The generative approach is particularly valuable because it can create visual representations for abstract or context-dependent sentences that might not have readily available corresponding images in traditional datasets.
  • Improved Noun and Subject Translation: The qualitative analysis suggests that visual context is especially effective in correctly identifying and translating concrete objects and entities, leading to more accurate noun phrase translations.

Key Results

The proposed multimodal translation framework demonstrated significant improvements over the NLLB baseline across four low-resource African languages.

  • Quantitative Gains: Average gains of up to 9.8% in BLEU score and 4.3% in METEOR score were observed.
  • Specific Language Performance: For Yoruba-English, the model achieved a BLEU of 23.96 (a 22.4% relative improvement over NLLB 3.3B). Hausa-English saw a BLEU of 37.21. Kanuri-English, a particularly low-resource language, showed substantial gains with a BLEU of 16.08.
  • Qualitative Improvements: Examples show the model correctly identifying "kite" from visual context (vs. "caterpillar" by baseline) and specifying locations like "balcony" (vs. generic "roof"). The LLM verification step also helped maintain semantic consistency with visual cues, as seen in the Tigrinya example where "summer" was correctly identified.

Limitations and Future Work

  • LLM as a Black Box: The LLM component, while enhancing supervision, doesn't truly "understand" the low-resource language semantics or its grammatical structures. This means that while metrics improve, capturing underlying linguistic nuances remains a challenge.
  • Alignment Assessment: Developing a generalized method for assessing image-text alignment is difficult, especially with imperfectly aligned real-world data. The authors acknowledge that their approach introduces unique error patterns due to this.
  • Baseline Limitations: The comparison is limited by the lack of existing multimodal models for low-resource languages. The text-only NLLB was used as a baseline, making it hard to definitively say if the architecture is optimal or if simpler multimodal approaches could yield similar results.
  • Future Work: The authors suggest that their framework's adaptability makes it promising for a wider range of low-resource languages, potentially aiding in digital preservation and cross-cultural communication. Further work could explore more sophisticated LLM integration and alignment assessment techniques.

Summary updated

BibTeX
@inproceedings{xiao2025from,
  author = {Bushi Xiao and Qian Shen and Daisy Zhe Wang},
  title = {{From Text to Multi-Modal: Advancing Low-Resource-Language Translation through Synthetic Data Generation and Cross-Modal Alignments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure