Empowering Low-Resource Language Translation: Methodologies for Bhojpuri-Hindi and Marathi-Hindi ASR and MT

Harpreet Singh Anand, Amulya Ratna Dash, Yashvardhan SharmaACL 2024

TL;DR

This paper tackles the challenge of building Automatic Speech Recognition (ASR) and Machine Translation (MT) systems for low-resource Indian languages like Bhojpuri and Marathi. They propose a cascaded approach, fine-tuning powerful pre-trained models like Whisper and NLLB, and introduce a novel method using sentence embeddings to select the best translation.

What Problem Does This Paper Address?

India is incredibly linguistically diverse, with over 100 official languages and countless dialects. This diversity poses a significant hurdle for Natural Language Processing (NLP) tasks like ASR and MT, especially for languages with limited digital resources. For millions of Indians who don't speak English, effective ASR and MT are crucial for accessing information and services. However, Indian languages often feature complex morphology, code-switching, and borrowing, making it difficult to build robust NLP systems. The IWSLT 2024 "Dialectal and Low-Resource Track" specifically challenges researchers to develop creative solutions for these languages, pushing beyond standard methods.

Methods

The authors employ a cascaded approach, meaning they build the system in stages: first ASR, then MT. For both Marathi-Hindi and Bhojpuri-Hindi, they leverage powerful pre-trained models and fine-tune them on the available datasets.

  • ASR:
    • Marathi-Hindi: They fine-tuned the whisper-small model (a multilingual variant) on two datasets: Common Voice and OpenSLR. They trained two separate models, whisper-ft-cv and whisper-ft-slr, and ultimately chose the latter as their primary submission due to better performance.
  • Bhojpuri-Hindi: For Bhojpuri ASR, they used a pre-trained vakyansh-wav2vec2-bhojpuri-bhom-60 model, which is specifically designed for low-resource Indic languages.
  • MT:
    • Marathi-Hindi: They fine-tuned a NLLB (No Language Left Behind) model (600M-distilled) on the Samanantar dataset for Marathi-to-Hindi translation.
  • Bhojpuri-Hindi: They used a NLLB (1.3B distilled) model for Bhojpuri-to-Hindi translation.

A key novel contribution is their choice of translation selection for the Marathi-Hindi task. After obtaining translations from both ASR models, they use the MuRIL model to generate sentence embeddings for both the original transcriptions and their translations. By comparing the cosine similarity between these embeddings, they select the translation that is most semantically similar to its corresponding transcription, aiming for higher accuracy.

Datasets

The authors utilized datasets provided by the IWSLT 2024 organizers:

  • OpenSLR (SLR64): This dataset contains transcribed high-quality audio of Marathi sentences.
  • Common Voice 11.0 (Marathi): This is a large, open-source, multilingual dataset used for fine-tuning the Whisper model.
  • Samanantar (Indic2Indic): This is described as the largest publicly available parallel corpus for Indic languages, used for fine-tuning the NLLB MT model for Marathi-Hindi.

Key Insights and Ideas

  • Leveraging Pre-trained Models is Crucial: The success of their systems hinges on fine-tuning large, pre-trained models (Whisper, NLLB, Wav2Vec2) that have already learned general linguistic patterns. This significantly reduces the need for massive amounts of language-specific data.
  • Dataset Choice Matters for ASR: Fine-tuning Whisper on the OpenSLR dataset yielded better ASR results for Marathi than fine-tuning on Common Voice, highlighting the importance of domain-specific data.
  • Semantic Similarity for Translation Selection: Using sentence embeddings (from MuRIL) to match transcriptions with their translations is an innovative way to pick the best output when multiple ASR hypotheses exist, going beyond simple word-level metrics.
  • Cascaded Approach Works: The paper demonstrates the effectiveness of a two-stage ASR-then-MT pipeline for these low-resource languages.

Key Results

  • Marathi-Hindi ASR: The whisper-ft-slr model achieved a Word Error Rate (WER) of 14.33917 and a Character Error Rate (CER) of 43.3628 on the evaluation dataset.
  • Marathi-Hindi ST (ASR+MT): The primary submission achieved a BLEU score of 21.3 and a chrF2 score of 48.1. While competitive, it trailed the best submission by a chrF2 score of 20.
  • Bhojpuri-Hindi ST (ASR+MT): The primary submission achieved a BLEU score of 12.9 and a chrF2 score of 41.1. This trailed the best submission by a chrF2 score of 8.4.

Limitations and Future Work

  • Performance Gap: The authors acknowledge that their systems trailed behind the best submissions in both the Marathi-Hindi and Bhojpuri-Hindi tracks.
  • Future Work: They plan to explore:
  • Data augmentation techniques to further boost performance.
  • Fine-tuning multiple pre-trained multilingual models.
  • Investigating more speech translation models specifically designed for low-resource Indian languages.

Summary updated

BibTeX
@inproceedings{anand2024empowering,
  author = {Harpreet Singh Anand and Amulya Ratna Dash and Yashvardhan Sharma},
  title = {{Empowering Low-Resource Language Translation: Methodologies for Bhojpuri-Hindi and Marathi-Hindi ASR and MT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure