A Character-Aware Encoder for Neural Machine Translation.

Zhen Yang, Wei Chen, Feng Wang, Bo XuCOLING 2016

TL;DR

This paper introduces a novel character-aware Neural Machine Translation (NMT) model that processes input text at the character level, eliminating the need for explicit word segmentation. This approach allows for more robust translation of languages like Chinese and significantly reduces the occurrence of "unknown words" in the output, achieving performance comparable to traditional word-based NMT systems.

What Problem Does This Paper Address?

Traditional Neural Machine Translation (NMT) models typically operate on word-level representations. A significant challenge with this approach is handling "out-of-vocabulary" (OOV) words, which are words not present in the model's predefined vocabulary. These OOV words are often mapped to a generic "unknown" token, leading to poor translation quality. Furthermore, word-based NMT systems struggle with languages that lack explicit word boundaries, such as Chinese, as they rely on whitespace or other delimiters to identify words. This paper aims to address both the OOV problem and the limitations of word segmentation by proposing a character-level NMT model.

Methods

The core innovation of this paper is a character-aware NMT encoder. Instead of processing words, the model takes sequences of characters as input. The encoder is designed to automatically compose word-level information from these character sequences. A key technique employed is row convolution, a method adapted from speech recognition. This allows the model to capture contextual information from neighboring characters.

The proposed model has two main components:

  • Context Computing: A bidirectional and concatenated row convolution layer is used to compute a context vector for each input character. This layer considers both past and future characters within a defined window, providing richer contextual information than traditional unidirectional approaches.
  • Character Composing: This layer combines the character embedding and its computed context vector to form a word-level representation. Two methods are proposed:
  • Forward character composing: A simple linear transformation combines the character and context embeddings.
  • Recurrent character composing: This method also incorporates the previous word-level representation, allowing for a more nuanced composition by considering potential interference from shared neighboring characters.

The decoder remains a standard RNN decoder, generating the target sentence word by word.

Datasets

The paper uses the following datasets:

  • Training Data: 2.3 million sentence pairs for Chinese to English translation.
  • Evaluation Benchmarks: IWSLT04, IWSLT05, IWSLT07, IWSLT08, MT08, and MT12 datasets are used for evaluating translation quality.

For the traditional word-based NMT baseline, a fixed vocabulary of 45k most frequent words for Chinese and 41k for English was used. For the character-aware NMT model, a character vocabulary of 7009 Chinese characters was built, covering all characters in the training data. The English vocabulary remained the same as the word-based model.

Key Insights and Ideas

  • Character-level processing bypasses word segmentation: The most significant insight is that by treating input as a sequence of characters, the model can directly handle languages like Chinese without needing a separate word segmentation step. This is a major advantage over traditional word-based NMT.
  • Row convolution effectively captures character context: The adaptation of row convolution proves effective in composing word-level representations from character sequences, demonstrating its utility beyond speech processing. The bidirectional and concatenated nature of their proposed row convolution further enhances context capture.
  • Character-aware NMT reduces unknown words: Even with a word-based decoder, the character-level encoder significantly reduces the number of unknown words in the output, leading to more fluent and complete translations.
  • Name entity translation improvement: The model shows surprising superiority in translating name entities, which are often rare and thus OOV in word-based systems. By breaking down names into characters, the model can correctly translate them.

Key Results

  • Comparable performance to word-based NMT: The proposed character-aware NMT model, particularly the "Character-aware-recurrent" version, achieves translation quality (measured by BLEU score) comparable to the traditional word-based RNNsearch model on Chinese-English translation tasks. For instance, on IWSLT04, both achieve around 50.01 BLEU.
  • Significant improvement over character-only NMT: The character-aware models (forward and recurrent) significantly outperform a baseline RNN that treats characters as input but doesn't effectively compose word-level representations (RNNsearch-Char).
  • Window size impact: The window size for row convolution is a critical hyperparameter. A window size of two yielded the best results, while a larger window size (four) led to convergence issues, suggesting that excessive context sharing can confuse the model.
  • Reduced OOV words: The paper states that the number of unknown words in output sentences is "dramatically decreased."
  • Name entity translation: Table 4 shows that the character-aware model can correctly translate names like "Naoko Suzuki" and "MS Amy Harris," whereas the word-based model outputs "<unk>" for these entities.

Limitations and Future Work

  • Word-based decoder: A primary limitation acknowledged is that the decoder still operates at the word level. The authors suggest that future work should investigate a setting where the target side is also represented as a character sequence.
  • Convergence with larger window sizes: The model struggles to converge with larger row convolution window sizes, indicating a need for further research into optimizing context aggregation.

Summary updated

Abstract

This article proposes a novel character-aware neural machine translation (NMT) model that views the input sequences as sequences of characters rather than words. On the use of row convolution (Amodei et al., 2015), the encoder of the proposed model composes word-level information from the input sequences of characters automatically. Since our model doesn’t rely on the boundaries between each word (as the whitespace boundaries in English), it is also applied to languages without explicit word segmentations (like Chinese). Experimental results on Chinese-English translation tasks show that the proposed character-aware NMT model can achieve comparable translation performance with the traditional word based NMT models. Despite the target side is still word based, the proposed model is able to generate much less unknown words.
BibTeX
@inproceedings{yang2016character,
  author = {Zhen Yang and Wei Chen and Feng Wang and Bo Xu},
  title = {{A Character-Aware Encoder for Neural Machine Translation.}},
  booktitle = {COLING},
  year = {2016},
}
Figure