TL;DR
This paper presents a Speech-to-Text Translation (STT) system designed to bridge communication gaps for low-resource Indian languages. By leveraging existing models and carefully curating data, the system aims to provide accurate English-to-Hindi, English-to-Bengali, and English-to-Tamil translations, paving the way for more inclusive digital communication.
What Problem Does This Paper Address?
Effective communication across diverse linguistic landscapes is a significant challenge, especially in regions like India with a multitude of languages. While advancements in Speech-to-Text Translation (STT) have been substantial for high-resource languages, many low-resource languages, particularly those in India, suffer from a critical lack of parallel data. This scarcity hinders the development of robust STT systems that can accurately translate spoken language into written text. The paper tackles this by aiming to build an efficient STT system for multiple low-resource Indian languages, specifically Hindi, Bengali, and Tamil, by creatively leveraging existing resources and addressing data disparities.
Methods
The system employs a cascaded approach, combining an Automatic Speech Recognition (ASR) module with a Machine Translation (MT) module.
- Audio Processing and Transcription: The system utilizes OpenAI's Whisper model for transcribing English audio. It also incorporates noise reduction and speech bandwidth enhancement using ResembleAI.
- Translation:
- For English-to-Hindi translation, the Helsinki-NLP model is used and fine-tuned.
- For English-to-Tamil and English-to-Bengali translation, Facebook's mBART model is employed and fine-tuned.
- Data Augmentation and Leveraging: The authors mention leveraging resources from related languages and exploring word-level translation resources to enhance accuracy. They also utilize TED talks, conference speeches, and parliamentary proceedings as data sources.
- Workflow: The system is structured into distinct modules: Input Module, Audio Processor and Transcription Module, Translation Module (with specific implementations for each language pair), and Output Module. The
translate.pyscript orchestrates these processes, whiletranscriber.pyhandles audio-to-text conversion andtranslator.py/fbtranslate.pymanage the text translation.
Datasets
The dataset used for training and evaluation consists of:
- Speeches from conferences and TED Talks.
- Corresponding transcriptions in English (source language).
- Translations in Hindi, Bengali, and Tamil (target languages).
The paper also references the IWSLT (International Conference on Spoken Language Translation) datasets and tasks, particularly the 2023 evaluation campaign, which focused on offline SLT.
Key Insights and Ideas
- Leveraging Pre-trained Models is Crucial: The success of the system hinges on the effective fine-tuning of powerful pre-trained models like Helsinki-NLP and mBART for low-resource languages. This demonstrates that even with limited specific data, transfer learning can yield significant improvements.
- Cascaded Approach for Low-Resource STT: Combining a robust ASR model (Whisper) with specialized MT models for each target language provides a practical and effective pipeline for tackling the STT challenge in low-resource scenarios.
- Importance of Data Curation: While the paper doesn't detail extensive new dataset creation, it highlights the use of diverse sources (conferences, TED Talks) and the need to address data disparities, suggesting that careful selection and preparation of existing data are vital.
- Modular Design for Scalability: The system's modular architecture (Input, Audio Processing, Translation, Output) makes it adaptable and easier to extend to other languages or integrate into larger applications.
Key Results
The paper presents SacreBLEU scores as a primary evaluation metric.
- English-to-Hindi: A SacreBLEU score of 24.21 was achieved using the Whisper/Helsinki model.
- English-to-Bengali: Scores of 14.18 (Whisper/Helsinki) and 16.18 (Whisper/mBART) were reported.
- English-to-Tamil: Scores of 7.1 (Whisper/Helsinki) and 10.79 (Whisper/mBART) were achieved.
The paper notes that for English-to-Bengali, the system is "barely crossing the baseline," indicating room for improvement. The baseline scores provided by the IWSLT INDIC Track are also mentioned: 5.23 for en-hi, 5.86 for en-bn, and 1.9 for en-ta. The system significantly outperforms these baselines, especially for Hindi and Bengali.
Limitations and Future Work
- Single Language Input: The current input module only supports one primary language. If an audio file contains multiple languages, the application ignores languages other than the primary one.
- Time to Produce Output: The time taken by the models to produce translations is a limitation.
- Bengali Performance: The system is barely crossing the baseline for English-to-Bengali translation, suggesting a need for further optimization.
Future work suggested includes exploring various optimizations and configurations to achieve faster results and improve performance, particularly for languages like Bengali. The authors aim to continue advancing ST technology for low-resource languages, fostering inclusivity and accessibility.
Summary updated