Chitranuvad: Adapting Multi-lingual LLMs for Multimodal Translation

Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Kumar Pokala, Ashish Kulkarni, Chandra Khatri, Abhinav Ravi, Shubham Agarwal

TL;DR

The authors introduce Chitranuvad, a vision-language model (VLM) tailored for Multimodal Machine Translation (MMT) into low-resource Indic languages (Hindi, Bengali, and Malayalam). By pairing a natively multilingual LLM backbone (Krutrim) with a visual encoder via a three-stage training pipeline, the model achieved state-of-the-art results for Hindi on the WAT 2024 English-to-Lowres shared task while remaining competitive across multiple tracks.

What Problem Does This Paper Address?

In multimodal translation, visual context helps resolve linguistic ambiguity—for example, deciding whether the word "court" refers to a tennis court or a legal institution. However, the majority of MMT research has historically targeted high-resource European languages.

Indic languages remain severely underrepresented in multimodal AI due to limited web-scale data in corpora like Common Crawl. Furthermore, most open-source "Indic" LLMs are merely adapted versions of English-centric models (such as LLaMA or Gemma), which struggle with nuanced linguistic representations. This paper tackles the challenge of building an image-grounded translation system capable of translating English text into Indic languages with high fidelity.

Methods

Chitranuvad adopts a LLaVA-style architecture, replacing the typical English-heavy backbone with Krutrim, a foundational multilingual LLM supporting 10 Indic languages.

  • Visual Encoding & Projection: Images are encoded using pre-trained vision encoders (evaluating CLIP ViT-L/14@336px and SigLIP-SO400M) and projected into the LLM embedding space via a multi-layer perceptron (MLP) adapter, producing 576 visual tokens within a 4,096-token context window.
  • Three-Stage Training Recipe:
  • Feature Alignment (Pre-training): The vision encoder and LLM are frozen; only the projection adapter is trained on image-caption pairs.
  • Instruction Tuning: The projection adapter and the LLM backbone are updated together on multi-turn instruction datasets, freezing only the vision encoder.
  • Task-Specific Fine-Tuning: Fine-tuning specifically for translation using Visual Genome data. The inputs are enriched with bounding-box coordinates and object tags detected using YOLOv8.

Datasets

The authors utilized a combination of translated academic datasets and task-specific benchmarks:

  • Stage 1 & 2: Pre-training used the 1.2M ShareGPT4V-PT dataset, translated into 10 Indic languages with an in-house translation engine. Instruction tuning incorporated translated versions of LLaVA-1.5-665K, the multi-task Cauldron dataset, and proprietary multimodal data.
  • Stage 3: The aligned multilingual Visual Genome datasets (Hindi, Bengali, and Malayalam) from the WAT2024 shared task, spanning ~29k training instances, ~1k validation, and ~1.6k test samples, plus a 1,400-instance Challenge set.

Key Insights and Ideas

  • Full Fine-Tuning Outperformed LoRA: Parameter-efficient fine-tuning (LoRA) significantly lagged behind full fine-tuning on the Visual Genome task (e.g., Hindi Challenge BLEU was 42.1 with LoRA vs. 53.0+ with full tuning), supporting the notion that LoRA learns less in cross-modal adaptation tasks.
  • Multilingual Training Prevents Catastrophic Forgetting: Fine-tuning specialized bilingual models (e.g., English-to-Hindi only) completely degraded translation performance on the other Indic languages (collapsing BLEU scores near zero). Training jointly on a mixture of Indic languages acted as a regularizer and preserved cross-lingual capabilities.
  • Native Multilingual Pre-training Beats Adaptation: In 0-shot translation benchmarks, Chitranuvad significantly outperformed PALO (a multilingual VLM based on Vicuna), even when Chitranuvad had only seen English multimodal data in Stage 1 and 2. The authors attribute this to Krutrim’s native multilingual foundation.
  • Back-Translation Didn’t Help: Adding reverse-direction translation data (Indic $\rightarrow$ English) during fine-tuning unexpectedly degraded automatic evaluation metrics.
  • The Persistent "Vision Paradox": Consistent with earlier MMT literature, visual context provided only marginal improvements over strong text-only baselines, suggesting that modern LLMs primarily rely on text signals unless visual disambiguation is strictly required.

Key Results

Competing under Team 007 at the WAT 2024 English-to-Lowres task across three tracks:

  • Multimodal Translation (MMT): Achieved SOTA on the Hindi Challenge set (54.7 BLEU post-deadline / 53.4 official vs. Volta’s 51.6 and BITS-P’s 52.1) and tied/led the Malayalam Test set (51.9 BLEU, 0.93 RIBES).
  • Text-Only Translation: Set SOTA on Hindi Challenge (55.3 BLEU post-deadline vs. 53.6 ODIAGEN) and Malayalam Test (51.7 BLEU vs. 46.6 ODIAGEN).
  • Image Captioning: Chitranuvad was the only submission capable of generating image captions across all three evaluated Indic languages (scoring 2.8 BLEU on Hindi Test and 1.8 BLEU on Bengali Test).

Limitations and Future Work

  • Frozen Vision Backbone: Throughout all stages, the vision encoder remained frozen. The authors highlight that unfreezing the vision encoder—potentially paired with attention pooling mechanisms like Perceiver Resamplers—could improve visual representation learning.
  • Limited Evaluated Languages: While the backbone supports 10 Indic languages, formal evaluation was constrained to the three shared-task languages (Hindi, Bengali, Malayalam).
  • Vision Modality Underutilization: The image modality still has a comparatively small effect on overall translation metrics compared to text context, leaving room for architectures that more aggressively condition translations on visual cues.

Summary updated

Abstract

In this work, we provide the system description of our submission as part of the English-to-Lowres Multimodal Translation Task at theWorkshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates Multilingual LLMand a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual tokenembeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all the three tracks (Image Captioning, Text-only and Multimodal translationtasks) for Indic languages (ie. English translation to Hindi, Bengali and Malyalam) and achieved SOTA results for Hindi in all of themon the Challenge set while remaining competitive for the other languages in the shared task.
BibTeX
@inproceedings{khan2024chitranuvad,
  author = {Shaharukh Khan and Ayush Tarun and Ali Faraz and Palash Kamble and Vivek Dahiya and Praveen Kumar Pokala and Ashish Kulkarni and Chandra Khatri and Abhinav Ravi and Shubham Agarwal},
  title = {{Chitranuvad: Adapting Multi-lingual LLMs for Multimodal Translation}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=gKi8wYXnFn},
}

Loading related content...

Figure