Abstract: Translating characters instead of words or word-fragments has the potential to simplify the processing pipeline for neural machine translation (NMT), and improve results by eliminating hyper-parameters and manual feature engineering. However, it results in longer sequences in which each symbol contains less information, creating both modeling and computational challenges. In this paper, we show that the modeling problem can be solved by standard sequence-to-sequence architectures of sufficient depth, and that deep models operating at the character level outperform identical models operating over word fragments. This result implies that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences. From this perspective, we evaluate several techniques for character-level NMT, verify that they do not match the performance of our deep character baseline model, and evaluate the performance versus computation time tradeoffs they offer. Within this framework, we also perform the first evaluation for NMT of conditional computation over time, in which the model learns which timesteps can be skipped, rather than having them be dictated by a fixed schedule specified before training begins.
Introduction
Most Neural Machine Translation (NMT) systems currently rely on subword-based models, such as Byte-Pair Encoding (BPE), which decompose words into common fragments. While BPE is efficient, it introduces manual hyperparameter tuning and can create linguistically nonsensical fragments, particularly in morphologically rich languages like Finnish or Czech.
The ideal approach is character-level NMT—processing text one letter at a time. Historically, this has been difficult because character sequences are significantly longer than word sequences, leading to two primary challenges: modeling long-range dependencies and the high computational cost of processing longer sequences. This paper revisits character-level NMT, demonstrating that depth is the key to overcoming these modeling hurdles.
Methods
The researchers at Google AI utilized a standard LSTM sequence-to-sequence model but significantly increased its capacity. The baseline model featured 6 bidirectional layers in the encoder and 8 layers in the decoder.
To address computational efficiency, the authors explored two temporal compression techniques:
- Fixed Stride Temporal Pooling: This method reduces sequence length by merging adjacent character representations into a single vector at specific layers.
- Learned Temporal Compression (HM-LSTM): Using a Hierarchical Multiscale LSTM, the model learns which characters are important enough to pass to higher layers. The authors modified the standard HM-LSTM with a "z-bias" to improve training stability.
Experimental Setup
The models were evaluated on four diverse language pairs from the WMT benchmarks:
- WMT15 Finnish-English (Fi-En): 2.1M pairs (High morphological complexity).
- WMT15 German-English (De-En): 4.5M pairs (Frequent compound words).
- WMT15 Czech-English (Cs-En): 14.8M pairs.
- WMT14 English-French (En-Fr): 39.9M pairs (Large-scale dataset).
Results and Discussion
The Impact of Depth
A critical finding is that character-level models have a steeper performance curve than BPE models. As model depth increases, the character-level models improve significantly faster than their subword counterparts, eventually surpassing them in quality across all tested language pairs.
Character models demonstrated a particular advantage in handling complex morphology and compound words, as seen in the German-English results. Qualitatively, these models were found to be more literal and less prone to omitting content from the source sentence.
Scaling with Data
The performance advantage of character-level modeling is most pronounced on small-to-medium datasets. As the corpus size grows into the tens of millions, BPE begins to close the gap, though character models maintain a slight edge up to the limits of the tested data.
Efficiency vs. Quality
While character models are roughly 8x slower to train than BPE models due to sequence length, compression techniques offer a viable trade-off. The HM-LSTM managed to save approximately 23% of computation while maintaining a BLEU score (31.2) that remained significantly higher than the BPE baseline (29.7).
Conclusion
This work demonstrates that character-level NMT can achieve superior translation quality when provided with sufficient model capacity. The primary remaining hurdle is computational cost. While learned compression techniques like HM-LSTM show promise, they are currently limited to unidirectional encoders. Future research should investigate whether these scaling laws for character-level modeling hold true for modern Transformer architectures.
Summary updated