Abstract
This paper explores the use of Neural Machine Translation (NMT) to translate between English-Tamil and Hindi-Tamil language pairs. By leveraging pre-trained subword embeddings (BPEmb), the researchers developed a system that handles the morphological complexity of Tamil while overcoming the "Out of Vocabulary" (OOV) challenges typical of low-resource languages.
Introduction
India is a land of immense linguistic diversity, with 23 constitutionally recognized languages. While a vast amount of valuable information is available online in English, only a small fraction of the Indian population is fluent in the language. This creates a significant information barrier for non-English speakers.
Manual translation is slow and costly, making Machine Translation (MT) an essential tool. However, Tamil presents unique challenges for MT:
- Morphological Richness: As an agglutinative language, Tamil words are formed by stringing together various morphemes, leading to a massive vocabulary where a single root can have thousands of variations.
- Syntactic Divergence: English follows a Subject-Verb-Object (SVO) order, whereas Tamil uses Subject-Object-Verb (SOV).
- Data Scarcity: There are relatively few high-quality parallel corpora available for training AI models in Indian languages compared to European ones.
Methodology
The researchers developed the MIDAS translator, a Neural Machine Translation system based on a Sequence-to-Sequence (Seq2Seq) architecture.
Technical Architecture
The model employs a 2-layer Bi-directional LSTM (Long Short-Term Memory) network as the encoder and a 2-layer LSTM as the decoder. A key feature is the Bahdanau Attention mechanism, which allows the model to focus on specific parts of the source sentence while generating each word of the translation, rather than compressing the entire sentence into a single fixed vector.
Subword Embeddings (BPEmb)
To address Tamil's morphological complexity, the system uses Byte-Pair Encoding Embeddings (BPEmb). Instead of whole-word units, it utilizes subword units combined with pre-trained embeddings from Wikipedia. This approach allows the model to interpret unknown words by analyzing their component parts (suffixes, prefixes, and roots), significantly reducing the "Out of Vocabulary" (OOV) problem.
Training and Datasets
The model was trained using datasets from VPT-IL, consisting of 1,223 English-Tamil pairs and 1,772 Hindi-Tamil pairs. The Adam optimizer was used with a learning rate of 0.001 and a BPE vocabulary size of 2000.
Results
The system's performance was evaluated using Precision and Recall metrics. Despite the limited training data, the MIDAS translator established the following baselines:
| Language Pair | % Precision | % Recall |
|---|---|---|
| English to Tamil | 26.97% | 37.98% |
| Tamil to Hindi | 25.18% | 27.24% |
The results demonstrate that subword-based NMT can effectively handle complex Indian language pairs even in low-resource scenarios. The authors found that a relatively simple model could achieve comparable results to more complex architectures if the input representations were high-quality and pre-trained.
Conclusion
The study highlights that subword units are superior to whole-word embeddings for agglutinative languages like Tamil. While the current model is limited by a small dataset, future work will focus on scaling the system with larger corpora and applying these techniques to code-switched languages (like "Hinglish" or "Tanglish") to improve conversational AI and content moderation.
Summary updated