JUCBNMT at WMT2018 News Translation Task: Character Based Neural Machine Translation of Finnish to English

Sainik Kumar Mahata, Dipankar Das, Sivaji BandyopadhyayEMNLP 2018

TL;DR

This paper introduces a character-based Neural Machine Translation (NMT) system for Finnish-to-English translation, participating in the WMT 2018 News Translation Shared Task. By focusing on character-level processing, the system aims to better handle morphological richness and rare words, achieving a BLEU score of 12.9.

What Problem Does This Paper Address?

Machine Translation (MT) systems aim to automatically translate text from one language to another. While Statistical Machine Translation (SMT) has been a dominant approach, Neural Machine Translation (NMT) has emerged as a more powerful paradigm, offering improved fluency and better handling of out-of-vocabulary (OOV) words. However, standard word-level NMT struggles with morphologically rich languages and rare words, often requiring massive parallel corpora to compensate. This paper addresses the challenge of translating morphologically complex languages like Finnish to English by proposing a character-based NMT approach, which can potentially learn morphological structures more effectively and mitigate the rare word problem.

Methods

The authors employ a character-based Neural Machine Translation system built upon the standard Encoder-Decoder architecture. Specifically, they utilize a Sequence-to-Sequence (Seq2Seq) model with a single Long Short-Term Memory (LSTM) layer for both the encoder and the decoder.

  • Preprocessing: The system involves several preprocessing steps:
  • Tokenization: Breaking down text into tokens (words, punctuation, numbers) using the NLTK toolkit.
  • Truecasing: Restoring original casing to improve data sparsity.
  • Cleaning: Removing excessively long sentences (more than 80 tokens) to ensure model stability.
  • Character-Level Encoding: Finnish and English characters are converted into one-hot vectors. The Finnish characters serve as input to the encoder, while the English characters (shifted by one time-step) are used as input for the decoder.
  • Encoder-Decoder Architecture:
    • The encoder uses a single LSTM layer to process the Finnish character sequence and encodes it into a fixed-length vector summarizing its meaning. The cell states are preserved for the decoder.
  • The decoder takes the encoder's cell states and the previous English character as input to predict the next English character. A Dense layer maps the decoder's output to the target English characters.
  • Training: The model is trained end-to-end by minimizing the log-likelihood. Key training parameters include a batch size of 128, 100 epochs, a softmax activation function, the rmsprop optimizer, categorical cross-entropy loss, and a learning rate of 0.001.

Datasets

The system was trained and evaluated using the parallel corpora provided by the WMT 2018 News Translation Shared Task organizers. This corpus consists of approximately 3.25 million Finnish-English sentence pairs.

Key Insights and Ideas

  • Character-level processing for morphological richness: The core idea is that by operating at the character level, the NMT model can implicitly learn the morphological structure of languages, which is particularly beneficial for morphologically rich languages like Finnish. This allows the model to construct words character by character, potentially handling unseen word forms more gracefully.
  • Mitigating rare word problem: Character-based models are expected to be more robust to rare words because they can construct them from known sub-word units (characters), unlike word-level models that might struggle with words not present in their vocabulary.
  • Simplicity of the architecture: The use of a single LSTM layer for both encoder and decoder, while potentially limiting, demonstrates a baseline approach to character-based NMT that can still yield competitive results.

Key Results

The system achieved a BLEU score of 12.9 on the Finnish-to-English translation task. Other evaluation metrics reported include:

  • BLEU Cased: 12.2
  • TER: 0.816
  • BEER 2.0: 0.448
  • Character TER: 0.770
  • Average Z (human evaluation): -0.404
  • Average % (human evaluation): 58.9

Limitations and Future Work

  • Limitations:
    • The system is described as "constrained," meaning it only used the data provided by the organizers for training.
  • The use of a single LSTM layer for both encoder and decoder might limit the model's capacity to capture complex linguistic patterns.
  • Future Work:
    • The authors plan to explore using more LSTM layers in their model to enhance its representational power.
  • They also intend to create an NMT model that takes words as input, rather than characters, and subsequently combine it with character-based embeddings.
  • Investigating various embedding schemes is also suggested to further improve translation quality.

Summary updated

BibTeX
@inproceedings{mahata2018jucbnmt,
  author = {Sainik Kumar Mahata and Dipankar Das and Sivaji Bandyopadhyay},
  title = {{JUCBNMT at WMT2018 News Translation Task: Character Based Neural Machine Translation of Finnish to English}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure