The RGNLP Machine Translation Systems for WAT 2018

Atul Kr. Ojha, Koel Dutta Chowdhury, Chao-Hong Liu, Karan Saxena

Abstract

The RGNLP team developed 42 machine translation systems for seven Indic languages to compete in the WAT 2018 Shared Task. This study evaluates the performance of traditional Phrase-Based Statistical Machine Translation (PBSMT) against Neural Machine Translation (NMT) in low-resource scenarios involving spoken language data. Our findings indicate that PBSMT significantly outperforms NMT in these contexts, particularly when dealing with informal syntax and high lexical diversity characteristic of subtitle data.

Introduction

While Neural Machine Translation (NMT) has achieved state-of-the-art results for high-resource language pairs, its efficacy in low-resource settings remains a subject of intense research. Indic languages, such as Hindi, Bengali, and Tamil, often lack the massive parallel corpora required for robust neural training. The RGNLP team participated in the "Indic Languages Multilingual Task" at WAT 2018 to assess how modern NMT stacks up against established Statistical Machine Translation (SMT) methods in the domain of spoken language (subtitles).

Methodology

The researchers developed 42 bidirectional systems, comprising 28 PBSMT and 14 NMT models, covering seven Indic languages paired with English.

PBSMT Configuration

The PBSMT systems were built using the Moses toolkit. Word alignment was performed with GIZA++, and language modeling utilized a combination of KenLM (5-gram) and SRILM (4-gram). This setup represents a highly optimized traditional statistical approach.

NMT Configuration

The NMT systems were implemented using OpenNMT-py, featuring a 2-layer Long Short-Term Memory (LSTM) network with an attention mechanism. Models were trained for 13 epochs using the Adam optimizer.

Pre-processing

A critical step was the use of a custom RGNLP tokenizer specifically designed for Indic scripts to ensure canonical Unicode representation, alongside standard Moses scripts for English true-casing and cleaning.

Datasets

The study utilized the WAT 2018 Indic Languages Multilingual Parallel Corpus, sourced from OpenSubtitles (Opus). The scale of the datasets varied significantly across language pairs, defining the "low-resource" nature of the task.

Table 1: Statistics of the parallel corpora used for training, tuning, and testing across seven Indic language pairs.

As shown in Table 1, the English-Singhalese pair provided the largest corpus (~523k sentences), while English-Telugu was the smallest (~23k sentences). Monolingual data was also incorporated, ranging from 2.8M sentences for English to 24k for Telugu.

Results and Discussion

The experimental results highlight a substantial performance gap between statistical and neural approaches in low-resource settings.

Figure 1: Comparison of BLEU scores between PBSMT and NMT systems across various English-Indic language pairs.

The Performance Gap

As illustrated in Figure 1, PBSMT consistently outperformed NMT across nearly all language pairs. The PBSMT model for English-Hindi was the star performer, achieving a BLEU score of 44.08. In contrast, the corresponding NMT system managed only approximately 21 BLEU—roughly half the performance of the statistical model. The most extreme failure was observed in the English-Singhalese pair, where NMT achieved a dismal 0.97 BLEU compared to a robust PBSMT score of approximately 18.

Analysis of NMT Limitations

The authors identified that NMT's poor performance in this context was driven by several factors:

  • Data Sensitivity: NMT models were highly sensitive to volume; smaller datasets led to "hallucinations" or total failure.
  • Out-of-Vocabulary (OOV) Issues: NMT struggled significantly with OOV words compared to PBSMT.
  • Syntactic Challenges: The informal syntax and short sentence lengths of spoken language data proved difficult for the specific LSTM-based NMT architecture used here.

Conclusion

Despite the industry-wide shift toward neural models, PBSMT remains the superior choice for low-resource Indic languages, especially within the spoken language domain. While PBSMT won on automatic metrics, human evaluation rankings suggest there is still a significant gap between machine output and human-level fluency. Future work will explore more advanced architectures, such as Transformers, and subword units (BPE) to better address the complex morphology of Indic scripts.

Summary updated

BibTeX
@inproceedings{ojha2018rgnlp,
  author = {Atul Kr. Ojha and Koel Dutta Chowdhury and Chao-Hong Liu and Karan Saxena},
  title = {{The RGNLP Machine Translation Systems for WAT 2018}},
  booktitle = {Wat@paclic},
  year = {2018},
}
Figure