Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation

Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala, Ashish Kulkarni, Chandra Khatri, Abhinav Ravi, Shubham Agarwal

Abstract

In this work, we provide the system description of our submission as part of the English to Lowres Multimodal Translation Task at the Workshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates a Multilingual LLM and a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual token embeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all three tracks (Image Captioning, Text-only, and Multimodal translation tasks) for Indic languages (English translation to Hindi, Bengali, and Malayalam) and achieved state-of-the-art (SOTA) results for Hindi in all of them on the Challenge set while remaining competitive for the other languages in the shared task.

Introduction

Machine translation often struggles with polysemy—words that have multiple meanings depending on context. For instance, the word "court" could refer to a sports venue or a judicial institution. Multimodal Machine Translation (MMT) addresses this by utilizing images as auxiliary information to ground linguistic symbols in visual reality. While MMT has advanced significantly for European languages, Indic languages—spoken by over a billion people—remain under-represented.

We present Chitranuvad ("Image Translate"), a multimodal Large Language Model (LLM) designed to translate English into low-resource Indic languages like Hindi, Bengali, and Malayalam. By integrating a natively multilingual backbone with a vision encoder, Chitranuvad achieves state-of-the-art (SOTA) results by effectively resolving linguistic ambiguities through visual context.

Methodology

Chitranuvad's architecture follows a modular design similar to LLaVA, consisting of three primary components:

  1. Backbone: The Krutrim LLM, a proprietary model natively trained on 10+ Indic languages, serves as the core reasoning engine.
  2. Vision Module: A Vision Transformer (ViT)—specifically CLIP or SigLIP—extracts visual features from images.
  3. The Bridge: An MLP adapter layer maps these visual features into the same mathematical space as the LLM’s text embeddings.
The three-stage training pipeline for Chitranuvad: Feature Alignment, Instruction Tuning, and Task-specific Finetuning.

The training process is structured into three distinct stages to ensure robust multimodal alignment and task performance:

  • Stage 1 (Feature Alignment): Training the adapter to connect images to text using 1.2 million image-text pairs.
  • Stage 2 (Instruction Tuning): Teaching the model to follow complex multimodal instructions, such as generating image descriptions in specific Indic languages.
  • Stage 3 (Task-specific Finetuning): Fine-tuning the entire system on specialized translation datasets. This stage includes an "object tagging" step where YOLOv8 identifies specific items in an image to provide explicit anchors for the model.

Datasets

The model was trained and evaluated using several key resources, including LLaVA-Pretrain, ShareGPT4V-PT, and the Cauldron dataset (translated into Indic languages). The core translation tasks utilized the Visual Genome datasets for Hindi, Bengali, and Malayalam, which provide images paired with English sentences and their corresponding Indic translations.

Results and Discussion

Chitranuvad achieved SOTA results for Hindi across all three tracks in the WAT2024 Challenge set. In text-only translation, it reached a BLEU score of 55.3. In the multimodal track, it achieved 54.7 for Hindi and 51.9 for Malayalam, demonstrating a clear lead over prior baselines like BITS-P and ODIAGEN.

Visual grounding example: Identifying objects like a tennis player helps resolve linguistic ambiguities in translation.

Key Insights

  • Multilingual Foundations: Using the natively multilingual Krutrim backbone allowed the model to outperform English-centric models in zero-shot Indic language tasks.
  • Full Finetuning: Contrary to the trend of using Parameter-Efficient Fine-Tuning (PEFT), the authors found that full parameter finetuning in the final stage was essential for superior translation quality.
  • Regularization via Language Mixing: To prevent "catastrophic forgetting," the researchers mixed data from all 10 supported Indic languages during training, which maintained high performance across the board.
  • Visual Grounding: Qualitatively, the model successfully used visual cues to distinguish between homonyms (e.g., "right" as a direction vs. "right" as correct), where text-only models frequently failed.

Conclusion

Chitranuvad demonstrates the power of grounding multilingual LLMs in visual context for low-resource language translation. While the visual modality's impact on BLEU scores remains a research bottleneck, the model's ability to resolve ambiguity is a significant step forward. Future work will focus on unfreezing the vision encoder and expanding support to the full spectrum of Indic languages supported by the Krutrim backbone.

Summary updated

Abstract

In this work, we provide the system description of our submission as part of the English to Lowres Multimodal Translation Task at the Workshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates Multilingual LLM and a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual token embeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all the three tracks (Image Captioning, Text only and Multimodal translation tasks) for Indic languages (ie. English translation to Hindi, Bengali and Malyalam) and achieved SOTA results for Hindi in all of them on the Challenge set while remaining competitive for the other languages in the shared task.
BibTeX
@inproceedings{khan2024chitranuvad,
  author = {Shaharukh Khan and Ayush Tarun and Ali Faraz and Palash Kamble and Vivek Dahiya and Praveen Pokala and Ashish Kulkarni and Chandra Khatri and Abhinav Ravi and Shubham Agarwal},
  title = {{Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=gKi8wYXnFn},
}
Figure