TL;DR
M-MiniGPT4 bridges the massive language gap in Vision Large Language Models (VLLMs) by expanding robust multimodal understanding to 11 languages. By combining machine-translated vision-language datasets with parallel text-only corpora across a novel three-stage training pipeline, it achieves state-of-the-art performance on multilingual visual reasoning benchmarks, even outperforming some later foundational models.
What Problem Does This Paper Address?
Most state-of-the-art open-source Vision Large Language Models (VLLMs) are heavily biased toward English, effectively excluding roughly 75% of the global population from benefiting from recent AI advancements. While text-only LLMs have rapidly expanded their language coverage, high-quality multimodal (image-text) data in non-English languages remains incredibly scarce.
Existing attempts to build multilingual VLLMs suffer from limited dataset sizes. Consequently, these models often excel at generating lengthy image descriptions but fail miserably at direct question-answering and complex, multi-step visual reasoning tasks. The authors aim to solve this data scarcity problem and build a model capable of true cross-lingual visual reasoning.
Methods
The authors build on the plug-and-play MiniGPT4 architecture, swapping out the original Vicuna backbone for Llama 3 to leverage its superior native multilingual text capabilities. To align the visual features with multiple languages, they introduce a structured three-stage training pipeline:
- Stage 1 (Modality Alignment): Standard alignment of vision and language features using large-scale English image-caption pairs.
- Stage 2 (Multilingual Understanding): Fine-tuning the model on a massive mixture of natively multilingual and machine-translated vision-language datasets to teach it how to understand images across different languages.
- Stage 3 (Multilingual Alignment): A novel step that introduces parallel text-only corpora (machine translation data) alongside translated text datasets. This stage tightens the alignment between languages, ensuring that the model's reasoning capabilities transfer seamlessly from English to other tongues.
Datasets
The project relies on a massive data translation effort using the No Language Left Behind (NLLB 1.3B) model to translate popular English datasets into 10 target languages (Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, Japanese, and Korean):
- Vision & Language (V&L) Datasets: Conceptual Captions, SBU, and LAION are used for Stage 1. For Stage 2, they introduce LAVAM (LLaVA-Instruct translated into 10 languages) and CI M (a translated version of the 58-million-pair Cambrian Image dataset), alongside the existing PALO dataset and a filtered version of the Wikipedia Image Text (WIT) dataset.
- Text Corpora: CT M (translated Cambrian Text) and MText (a combination of the native Flores translation pairs and the human-translated XStoryCloze dataset) are utilized in Stage 3.
- Evaluation Benchmark: The authors translated the popular MMMU visual reasoning benchmark into the 10 target languages. They validated the translation quality using back-translation to ensure no significant information loss occurred during the process.
Key Insights and Ideas
- Translation is a Powerful Bootstrap: Simply translating high-quality English instruction datasets into target languages yields massive boosts in multilingual performance without degrading original English capabilities.
- Text-Only Parallel Data Aids Multimodal Alignment: Natively parallel text-only data (like translation pairs in MText) acts as a cross-lingual bridge. Introducing it in the final stage helps the model ground visual concepts across different languages, even when parallel image data is missing.
- Scale and Diversity Unlock Reasoning: Moving beyond simple image-caption pairs to massive, diverse instruction-tuning datasets (like Cambrian Image) is what allows the model to transition from simple image description to complex, multi-step visual reasoning.
Key Results
- On the MMMU Multi benchmark, M-MiniGPT4 achieves 33.45% accuracy, vastly outperforming its direct predecessor PALO (13.12%).
- Remarkably, M-MiniGPT4 outperforms the foundational Qwen-VL 2.5 (25.46%) on the multilingual benchmark, despite Qwen-VL 2.5 being a much larger model released later. However, Qwen-VL 2.5 still maintains an edge on the standard English MMMU due to its larger instruction-tuning scale.
- Ablation studies confirmed that the combination of translated image data (CI M) in Stage 2 and parallel text data (MText) in Stage 3 yields the absolute best performance.
Limitations and Future Work
Despite its strong performance, M-MiniGPT4 faces a few key limitations:
- Translation Nuances: Relying heavily on machine translation means the datasets might miss subtle cultural contexts and linguistic nuances unique to native datasets.
- Resource Disparity: Performance and translation quality remain noticeably better for high-resource languages (e.g., Spanish, French) than for lower-resource ones (e.g., Bengali, Urdu).
- Language Coverage: While 11 languages is a major step forward, it still represents only a tiny fraction of the world's languages.
The authors have open-sourced their models, code, and translated datasets to encourage the community to expand on this work, particularly for low-resource languages.
Summary updated