LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages

Junchen Li, Qing Yang, Bojian Jiang, Shaolin Zhu, Qingxuan SunAAAI 2025 · DOI

TL;DR

This paper introduces LRM-LLaVA, a novel multilingual large language-vision model (LVLM) designed to bridge the modality gap and improve performance on low-resource languages. By employing a two-stage training strategy with specialized multilingual task instructions and a cross-modal regularizer, LRM-LLaVA significantly enhances cross-lingual understanding without sacrificing English capabilities.

What Problem Does This Paper Address?

Large language-vision models (LVLMs) have shown impressive capabilities in understanding and generating content from both text and images, particularly in English. However, their performance plummets when applied to non-English languages, especially those with limited data. This disparity, termed the "modality gap," arises from the difficulty in aligning visual features with textual representations across different languages, exacerbated by the scarcity of high-quality multilingual training data. Existing LVLMs, often trained primarily on English data, struggle to generalize effectively to other linguistic contexts.

Methods

LRM-LLaVA is built upon a four-component architecture:

  1. Visual Encoder: A pre-trained ViT-L/14 model extracts visual features.
  2. Multilingual Large Language Model: A Vicuna-13B model serves as the backbone for text processing and response generation.
  3. Vision-Text Representation Projector: A two-layer MLP bridges the visual and textual modalities by transforming visual features into a format compatible with the language model.
  4. Cross-Modal Regularizer: This component is key to reducing the modality gap. It utilizes three multilingual task instructions—Monolingual Vision Question Answering (MVQA), English-Pivot Vision Question Answering (EP-VQA), and Bilingual Vision Question Answering (BVQA)—to encourage consistent representations across modalities.

The training process is divided into two stages:

  • Pre-training: Focuses on aligning visual and multilingual features using a multilingual task instruction. The visual encoder and LLM are frozen, and only the projector is trained.
  • Instruction Fine-tuning: Aims to improve the model's ability to follow instructions. The LLM and projector are trained using three multilingual task instructions on diverse data, enhancing generalization.

A novel aspect is the construction of a multilingual visual question answering dataset by translating existing English datasets and creating specific task instructions.

Datasets

  • Training Data:
    • Over 120M English pre-trained image-text alignment data (ShareGPT4V-PT).
  • Over 120M English instruction fine-tuning datasets (LLaVA-1.5 mixtures, ChartQA, GeoQA, DocVQA, DVQA, AI2D).
  • Translated versions of these datasets into non-English languages, forming the core of the multilingual training data.
  • Benchmarks:
    • Four English benchmarks (MME, MMBench, POPE, SEED-Bench) were translated into 10 languages to create new multilingual evaluation benchmarks.
  • The paper also uses existing multilingual benchmarks for evaluation.

Key Insights and Ideas

  • The Modality Gap is Real and Significant: The paper strongly demonstrates that current LVLMs, despite their multilingual capabilities, suffer a substantial performance drop on non-English tasks due to the inherent difficulty in aligning visual and textual information across languages.
  • Multilingual Task Instructions are Crucial: The introduction of MVQA, EP-VQA, and BVQA instructions, especially when used in a structured two-stage training process, is shown to be highly effective in bridging the modality gap. EP-VQA, in particular, leverages English as a pivot to improve cross-lingual understanding.
  • Data Augmentation via Translation is Viable: The paper successfully creates multilingual training data and benchmarks by translating existing English resources, offering a practical solution for low-resource scenarios.
  • Cross-Modal Regularization Works: The proposed cross-modal regularizer, guided by the multilingual task instructions, demonstrably improves the alignment between visual and textual feature representations, as visualized by T-SNE plots.

Key Results

  • LRM-LLaVA achieves competitive performance compared to other multilingual LVLMs with similar parameter counts.
  • It significantly outperforms previous models on non-English tasks, with improvements noted even for low-resource languages like Croatian and Czech.
  • The model shows strong performance on English tasks, indicating that multilingual training does not degrade English capabilities.
  • Qualitative examples (Figure 4) show LRM-LLaVA providing more accurate and detailed descriptions for Czech questions compared to LLaVA-1.5.
  • An ablation study (Table 3) confirms the importance of each task instruction, with MVQA and EP-VQA showing significant contributions.

Limitations and Future Work

  • Translation Quality: While the paper uses Google Translate and GPT-40 and validates their reliability, the inherent limitations of machine translation for complex or nuanced content remain a potential source of error.
  • Data Size: The paper explores data synthesis ratios, suggesting 100% non-English data integration is optimal. However, the absolute amount of data and its impact on extremely low-resource languages could be further investigated.
  • Model Scale: While larger models (13B vs. 7B) show greater improvement with the proposed method, exploring even larger models or more efficient architectures could yield further gains.
  • Future Work: The authors suggest further exploration into optimizing the data synthesis ratio and investigating the impact of model scale. The success of LRM-LLaVA opens avenues for developing more robust and equitable LVLMs for a wider range of languages.

Summary updated

BibTeX
@inproceedings{li2025lrm,
  author = {Junchen Li and Qing Yang and Bojian Jiang and Shaolin Zhu and Qingxuan Sun},
  title = {{LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34623},
}
Figure