Revisiting Character-Based Neural Machine Translation with Capacity and Compression

Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, Wolfgang Macherey

Abstract

Translating characters instead of words or word-fragments has the potential to simplify the processing pipeline for neural machine translation (NMT), and improve results by eliminating hyper-parameters and manual feature engineering. However, it results in longer sequences in which each symbol contains less information, creating both modeling and computational challenges. In this paper, we show that the modeling problem can be solved by standard sequence-to-sequence architectures of sufficient depth, and that deep models operating at the character level outperform identical models operating over word fragments. This result implies that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences. From this perspective, we evaluate several techniques for character-level NMT, verify that they do not match the performance of our deep character baseline model, and evaluate the performance versus computation time tradeoffs they offer. Within this framework, we also perform the first evaluation for NMT of conditional computation over time, in which the model learns which timesteps can be skipped, rather than having them be dictated by a fixed schedule specified before training begins.

Introduction

Most modern NMT systems rely on subword tokenization, like Byte-Pair Encoding (BPE), to handle large vocabularies and rare words. While effective, BPE is a heuristic pre-processing step that requires manual hyperparameter tuning and can produce linguistically nonsensical fragments (e.g., fling → fl + ing).

Character-level models are an attractive alternative because they eliminate these pre-processing steps and give the model access to all information in the source and target sequences. However, they face two major hurdles: modeling challenges due to longer sequences with less semantic information per symbol, and computational challenges due to higher per-layer costs and quadratic attention costs.

Capacity and Scaling

A key finding of this research is that character-level models have a much steeper performance curve than BPE models as model capacity increases. While BPE models might start ahead at shallow depths, character models quickly overtake them as the architecture scales. This suggests that previous research finding parity between the two approaches may have been limited by insufficient model depth.

BLEU scores for character-level vs BPE models across four language pairs as model depth increases.

The gains from character-level modeling are most pronounced in morphologically rich languages like Finnish and German. Characters allow the model to learn the internal structure of complex words that BPE might fragment poorly. Qualitative analysis also shows that character models are more faithful to the source text, being significantly less likely to "hallucinate" translations compared to BPE models, which sometimes struggle with lexical choices for fragmented words.

Efficiency and Temporal Compression

The primary drawback of character-level models remains their computational cost. Without compression, character models are roughly 8x slower to train than BPE models. As shown in the training time analysis, the cost scales linearly with the number of layers, but with a significantly higher constant factor for characters.

Training time per sentence (milliseconds) versus the number of layers for character and BPE models.

To address this, the authors explore temporal compression—reducing the number of hidden states as information flows through the encoder. They test two main approaches: fixed stride temporal pooling and learned temporal compression using a modified Hierarchical Multiscale LSTM (HM-LSTM). The learned HM-LSTM compression was able to save approximately 35% of encoder computations while maintaining performance nearly identical to the uncompressed character baseline. This suggests that learned compression is a viable path toward making character models practical.

Conclusion

Character-level NMT can consistently outperform subword-based models if given sufficient model depth. While computational cost remains a challenge, learned temporal compression offers a viable path toward making these high-quality models practical. The results suggest that the NMT community should focus on developing architectures that can approximate the quality of deep character-level models while matching the efficiency of subword-level processing.

Summary updated

Abstract

Translating characters instead of words or word-fragments has the potential to simplify the processing pipeline for neural machine translation (NMT), and improve results by eliminating hyper-parameters and manual feature engineering. However, it results in longer sequences in which each symbol contains less information, creating both modeling and computational challenges. In this paper, we show that the modeling problem can be solved by standard sequence-to-sequence architectures of sufficient depth, and that deep models operating at the character level outperform identical models operating over word fragments. This result implies that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences. From this perspective, we evaluate several techniques for character-level NMT, verify that they do not match the performance of our deep character baseline model, and evaluate the performance versus computation time tradeoffs they offer. Within this framework, we also perform the first evaluation for NMT of conditional computation over time, in which the model learns which timesteps can be skipped, rather than having them be dictated by a fixed schedule specified before training begins.
BibTeX
@inproceedings{cherry2018revisiting,
  author = {Colin Cherry and George Foster and Ankur Bapna and Orhan Firat and Wolfgang Macherey},
  title = {{Revisiting Character-Based Neural Machine Translation with Capacity and Compression}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=r1bMIzzdbS},
}
Figure