Neural Machine Translation with Byte-Level Subwords

Changhan Wang, Kyunghyun Cho, Jiatao Gu

Abstract

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.

Introduction

Standard Neural Machine Translation (NMT) models typically rely on subword tokenization, such as Byte-Pair Encoding (BPE), at the character level. While effective, this approach faces significant challenges when dealing with "character-rich" languages like Chinese or Japanese, or when processing noisy text. In these scenarios, the massive number of unique characters leads to bloated vocabularies where rare characters occupy valuable slots despite being seldom seen during training.

Furthermore, character-based models are inherently script-bound. A model trained on a Latin script (e.g., English) cannot easily adapt to a disjoint script like Sinhala. While raw byte-level vocabularies offer a "universal" solution, they often result in excessively long sequences that are computationally prohibitive. This research proposes Byte-Level BPE (BBPE) as a middle ground, combining the efficiency of subwords with the universality and compactness of bytes.

Methodology: Byte-Level BPE

The core innovation is the shift from characters to bytes as the atomic unit of tokenization. Instead of initiating the BPE merging process with Unicode characters, the process starts with the 256 possible values of a UTF-8 byte.

Byte-level representation of Japanese characters showing the mapping from characters to UTF-8 byte sequences.

Tokenization and Contextualization

Text is first converted to UTF-8 byte sequences, and BPE is applied to learn variable-length byte $n$-grams. However, because individual bytes or partial sequences lack the semantic clarity of full characters, the authors introduce a "contextualization" step. By inserting a Depth-Wise Convolutional layer or a Bidirectional GRU (Bi-GRU) immediately after the embedding layer, the model can better learn character boundaries and disambiguate byte sequences.

Decoding

To ensure valid output, a linear-time Dynamic Programming (DP) algorithm is used during inference. This prevents the model from predicting invalid byte sequences that do not form valid Unicode characters, ensuring the output is always recoverable.

Key Results and Insights

The researchers evaluated BBPE across bilingual (En-De, Ja-En, Si-En) and multilingual (58 languages to English) tasks, comparing it against standard Transformer baselines.

Performance and Efficiency

BBPE allows for significantly more compact vocabularies. On English-German (En-De), a BBPE model with a 2K vocabulary achieved a BLEU score of 26.98, nearly matching a standard 32K BPE model (27.31 BLEU). In character-rich contexts like Japanese-English (Ja-En), BBPE 4K+ outperformed the BPE 16K+ baseline in several test sets.

The Importance of Context

A critical finding is that raw byte embeddings are too granular for standard Transformers to handle efficiently. As shown in the experimental results, adding a contextualization layer provides a substantial performance boost (over 4% in BLEU) across various vocabulary sizes.

Comparison of translation quality (BLEU) with and without contextualization layers (Convolutional and Bi-GRU) across different tokenization methods.

Universal Script and Transfer Learning

In multilingual settings, BBPE maximizes parameter sharing. Since all languages share the same underlying byte distributions, BBPE enables zero-shot transfer. A model pretrained on 58 languages and fine-tuned on Sinhala-English (a script not seen during pretraining) achieved 9.0 BLEU, significantly higher than the 7.2 BLEU of a model trained from scratch.

Conclusion

BBPE offers a robust alternative to character-level subwords, providing high vocabulary efficiency and enabling cross-script transfer learning. While the resulting sequences are longer than character-BPE, they remain more efficient than pure byte sequences. Future work will focus on optimizing segmentation to further reduce sequence length and exploring broader many-to-many translation settings.

Summary updated

Abstract

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.
BibTeX
@inproceedings{wang2020neural,
  author = {Changhan Wang and Kyunghyun Cho and Jiatao Gu},
  title = {{Neural Machine Translation with Byte-Level Subwords}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=rTfZA6VHpk5},
}
Figure