Abstract
In the landscape of Machine Translation (MT), Neural Machine Translation (NMT) has become the standard for high-resource language pairs. However, its efficacy in low-resource environments, such as those involving Indic languages, remains a subject of investigation. This paper summarizes the RGNLP team's participation in the WAT 2018 Indic Languages Multilingual Task, where they compared Phrase-Based Statistical Machine Translation (PBSMT) and NMT across seven Indic languages. The results indicate that traditional statistical methods often maintain a performance edge in data-scarce scenarios.
Introduction: The Low-Resource Hurdle
While NMT dominates rich-resource pairs like English-German, it is notoriously data-hungry. Indic languages—including Hindi, Telugu, and Malayalam—frequently lack the massive parallel corpora required to train robust neural models. The RGNLP team addressed this "low-resource hurdle" by benchmarking the established Moses-based PBSMT against a baseline NMT system using the Indic Languages Multilingual Parallel Corpus.
Methodology
The researchers developed 42 bidirectional MT systems using two primary frameworks:
- Phrase-Based SMT (PBSMT): Built using the Moses toolkit, this system utilized GIZA++ for word alignment. Language models (4-gram and 5-gram) were constructed using KenLM and SRILM to capture statistical phrase patterns.
- Neural MT (NMT): Implemented via OpenNMT-py, the NMT systems used a 2-layer Long Short-Term Memory (LSTM) network with 500 hidden units. Default hyperparameters were maintained to establish a baseline for "out-of-the-box" NMT performance in low-resource settings.
A critical component of the workflow was a custom tokenizer developed by the RGNLP team. This tool was designed specifically for Indic scripts to ensure canonical Unicode representation, alongside standard cleaning and true-casing for English text.
Data and Evaluation
The study utilized the Indic Languages Multilingual Parallel Corpus, primarily sourced from OpenSubtitles (Opus). The dataset covered Hindi, Bengali, Malayalam, Tamil, Telugu, Singhalese, and Urdu.
As shown in the data statistics, volume varied significantly across pairs. English-Singhalese was the largest (~521k sentences), while English-Telugu was the smallest (~22k sentences). Systems were evaluated using automatic metrics (BLEU, RIBES, AMFM) and human adequacy assessments for English-Hindi pairs.
Results and Analysis
The evaluation revealed a consistent performance gap between the two paradigms.
PBSMT Dominance in Low-Resource Settings
PBSMT models achieved the highest scores across all automatic metrics for English into Telugu, Hindi, Bengali, and Tamil. The English-Hindi PBSMT system was the top performer, reaching 44.08 BLEU. In contrast, NMT performance peaked at approximately 21 BLEU for the same pair—less than half the score of its statistical counterpart.
Qualitative Insights
- Data Sensitivity: NMT performance was strictly tied to data volume. Even in the relatively larger English-Singhalese dataset, NMT lagged behind PBSMT.
- Robustness to OOV: PBSMT proved more resilient to Out-of-Vocabulary (OOV) words. While NMT often "hallucinated" or suffered from catastrophic word-order failures when encountering unknown terms, PBSMT was more likely to preserve the overall sentence structure.
- Human Evaluation: In human adequacy rankings for English-Hindi, the RGNLP systems secured the 3rd rank among all participants, validating the competitive nature of their statistical approach.
Conclusion and Future Directions
The findings underscore the challenges of applying standard NMT architectures to low-resource Indic languages. The structural differences between English (SVO) and Indic languages (typically SOV) exacerbate the learning difficulty when training data is limited. Future work should explore more advanced architectures, such as Transformers, or data augmentation techniques like back-translation to bridge the performance gap currently filled by traditional statistical methods.
Summary updated