TL;DR
This paper details the submission of the "Translation Techies" team to the ACL 2022 Shared Task on Machine Translation for Dravidian languages. They adapted a baseline Transformer model from the OpenNMT framework, applied tokenization using the IndicNLP library, and evaluated performance using BLEU scores, highlighting the challenges and potential of MT for low-resource Dravidian languages.
What Problem Does This Paper Address?
India, a linguistically diverse country, faces a significant challenge in bridging language barriers, especially for its numerous human languages. While English dominates online content, translating between India's many languages, particularly the morphologically rich and script-diverse Dravidian languages, requires robust and efficient machine translation (MT) systems. This paper addresses the need for constructive and precise algorithms that can minimize human intervention in translating these languages, aiming to facilitate knowledge dissemination and end-to-end speech translation. The task specifically focuses on low-resource Dravidian languages, which are often difficult to handle due to their complex morphology and unique scripts.
Methods
The team employed a Transformer architecture as their baseline model, leveraging the OpenNMT framework. This choice is significant as the Transformer model, with its self-attention mechanism, has become the de facto standard for state-of-the-art MT. They utilized the IndicNLP library for tokenization, a crucial pre-processing step that breaks down text into smaller units. This was applied to the input data before feeding it into the Transformer model. The evaluation metric used was BLEU (Bilingual Evaluation Understudy), a standard for assessing the quality of machine-translated text by comparing it to human-generated reference translations.
Datasets
The paper describes the use of datasets for five language pairs involving Kannada as the source language and Malayalam, Sanskrit, Tamil, Telugu, and Tulu as target languages. The sizes of these datasets (training, validation, and testing) are provided in Table 1.
- Kannada-Malayalam: 90,974 (train), 2,000 (valid), 2,000 (test)
- Kannada-Sanskrit: 9,470 (train), 1,000 (valid), 1,000 (test)
- Kannada-Tamil: 88,813 (train), 2,000 (valid), 2,000 (test)
- Kannada-Telugu: 88,503 (train), 2,000 (valid), 2,000 (test)
- Kannada-Tulu: 8,300 (train), 1,000 (valid), 1,000 (test)
The paper notes that the datasets for Sanskrit and Tulu were smaller, yet yielded better results, which is discussed further in the "Key Insights and Ideas" section.
Key Insights and Ideas
- Tokenization Matters: Initially, the baseline model without tokenization resulted in a high number of "unknown" tokens in the output. Applying tokenization using the IndicNLP library significantly improved results, leading to more translated words and fewer unknown tokens. This highlights the importance of appropriate pre-processing for morphologically rich languages.
- Dataset Size vs. Performance: Surprisingly, the model achieved better BLEU scores for Sanskrit and Tulu, despite having smaller training datasets. The authors attribute this to a higher overlap of sentences and words between the training and test sets for these pairs, suggesting that dataset quality and similarity can be as crucial as sheer size for low-resource languages.
- Dravidian Languages' Complexity: The paper acknowledges the inherent difficulty in translating Dravidian languages due to their rich morphology and unique scripts, underscoring the ongoing challenge in developing effective MT systems for them.
Key Results
The paper reports BLEU scores for the five language pairs:
- Kannada-Malayalam: 0.0729
- Kannada-Sanskrit: 0.7482
- Kannada-Tamil: 0.0798
- Kannada-Telugu: 0.1242
- Kannada-Tulu: 0.6149
While the scores vary across pairs, the highest scores were achieved for Kannada-Sanskrit and Kannada-Tulu. The authors note that these scores, particularly for Telugu, do not provide deep insights due to the small and limited datasets, but they represent a starting point for improvement.
Limitations and Future Work
The authors acknowledge that the current results are limited by the size and quality of the available datasets. They suggest several avenues for future work:
- Enhancing Dataset Quality: Improving the quality of existing datasets and potentially augmenting them.
- Advanced Pre-processing: Implementing more sophisticated pre-processing techniques, such as byte-pair encoding (BPE) and data augmentation, to better handle word variations and improve translation accuracy.
- Subword Tokenization: Exploring subword tokenization techniques, which have proven effective in addressing issues with low-resource language pairs.
- Model Enhancement: Further improving the Transformer model architecture or exploring other advanced MT techniques.
The ultimate goal is to develop more efficient and precise MT systems to bridge the language divide and facilitate knowledge dissemination in Dravidian languages.
Summary updated