TL;DR
The authors introduce Chitranuvad, a vision-language model (VLM) tailored for Multimodal Machine Translation (MMT) into low-resource Indic languages (Hindi, Bengali, and Malayalam). By pairing a natively multilingual LLM backbone (Krutrim) with a visual encoder via a three-stage training pipeline, the model achieved state-of-the-art results for Hindi on the WAT 2024 English-to-Lowres shared task while remaining competitive across multiple tracks.
What Problem Does This Paper Address?
In multimodal translation, visual context helps resolve linguistic ambiguity—for example, deciding whether the word "court" refers to a tennis court or a legal institution. However, the majority of MMT research has historically targeted high-resource European languages.
Indic languages remain severely underrepresented in multimodal AI due to limited web-scale data in corpora like Common Crawl. Furthermore, most open-source "Indic" LLMs are merely adapted versions of English-centric models (such as LLaMA or Gemma), which struggle with nuanced linguistic representations. This paper tackles the challenge of building an image-grounded translation system capable of translating English text into Indic languages with high fidelity.
Methods
Chitranuvad adopts a LLaVA-style architecture, replacing the typical English-heavy backbone with Krutrim, a foundational multilingual LLM supporting 10 Indic languages.
- Visual Encoding & Projection: Images are encoded using pre-trained vision encoders (evaluating CLIP ViT-L/14@336px and SigLIP-SO400M) and projected into the LLM embedding space via a multi-layer perceptron (MLP) adapter, producing 576 visual tokens within a 4,096-token context window.
- Three-Stage Training Recipe:
- Feature Alignment (Pre-training): The vision encoder and LLM are frozen; only the projection adapter is trained on image-caption pairs.
- Instruction Tuning: The projection adapter and the LLM backbone are updated together on multi-turn instruction datasets, freezing only the vision encoder.
- Task-Specific Fine-Tuning: Fine-tuning specifically for translation using Visual Genome data. The inputs are enriched with bounding-box coordinates and object tags detected using YOLOv8.
Datasets
The authors utilized a combination of translated academic datasets and task-specific benchmarks:
- Stage 1 & 2: Pre-training used the 1.2M ShareGPT4V-PT dataset, translated into 10 Indic languages with an in-house translation engine. Instruction tuning incorporated translated versions of LLaVA-1.5-665K, the multi-task Cauldron dataset, and proprietary multimodal data.
- Stage 3: The aligned multilingual Visual Genome datasets (Hindi, Bengali, and Malayalam) from the WAT2024 shared task, spanning ~29k training instances, ~1k validation, and ~1.6k test samples, plus a 1,400-instance Challenge set.
Key Insights and Ideas
- Full Fine-Tuning Outperformed LoRA: Parameter-efficient fine-tuning (LoRA) significantly lagged behind full fine-tuning on the Visual Genome task (e.g., Hindi Challenge BLEU was 42.1 with LoRA vs. 53.0+ with full tuning), supporting the notion that LoRA learns less in cross-modal adaptation tasks.
- Multilingual Training Prevents Catastrophic Forgetting: Fine-tuning specialized bilingual models (e.g., English-to-Hindi only) completely degraded translation performance on the other Indic languages (collapsing BLEU scores near zero). Training jointly on a mixture of Indic languages acted as a regularizer and preserved cross-lingual capabilities.
- Native Multilingual Pre-training Beats Adaptation: In 0-shot translation benchmarks, Chitranuvad significantly outperformed PALO (a multilingual VLM based on Vicuna), even when Chitranuvad had only seen English multimodal data in Stage 1 and 2. The authors attribute this to Krutrim’s native multilingual foundation.
- Back-Translation Didn’t Help: Adding reverse-direction translation data (Indic $\rightarrow$ English) during fine-tuning unexpectedly degraded automatic evaluation metrics.
- The Persistent "Vision Paradox": Consistent with earlier MMT literature, visual context provided only marginal improvements over strong text-only baselines, suggesting that modern LLMs primarily rely on text signals unless visual disambiguation is strictly required.
Key Results
Competing under Team 007 at the WAT 2024 English-to-Lowres task across three tracks:
- Multimodal Translation (MMT): Achieved SOTA on the Hindi Challenge set (54.7 BLEU post-deadline / 53.4 official vs. Volta’s 51.6 and BITS-P’s 52.1) and tied/led the Malayalam Test set (51.9 BLEU, 0.93 RIBES).
- Text-Only Translation: Set SOTA on Hindi Challenge (55.3 BLEU post-deadline vs. 53.6 ODIAGEN) and Malayalam Test (51.7 BLEU vs. 46.6 ODIAGEN).
- Image Captioning: Chitranuvad was the only submission capable of generating image captions across all three evaluated Indic languages (scoring 2.8 BLEU on Hindi Test and 1.8 BLEU on Bengali Test).
Limitations and Future Work
- Frozen Vision Backbone: Throughout all stages, the vision encoder remained frozen. The authors highlight that unfreezing the vision encoder—potentially paired with attention pooling mechanisms like Perceiver Resamplers—could improve visual representation learning.
- Limited Evaluated Languages: While the backbone supports 10 Indic languages, formal evaluation was constrained to the three shared-task languages (Hindi, Bengali, Malayalam).
- Vision Modality Underutilization: The image modality still has a comparatively small effect on overall translation metrics compared to text context, leaving room for architectures that more aggressively condition translations on visual cues.
Summary updated