Neural Machine Translation with Byte-Level Subwords

Changhan Wang, Kyunghyun Cho, Jiatao Gu

Abstract

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords, or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese can unnecessarily take up vocabulary slots and limit compactness. Representing text at the level of bytes using the 256-byte set as a vocabulary is a potential solution, but high computational costs have prevented its wide adoption. In this paper, we investigate byte-level subwords, specifically Byte-Level Byte Pair Encoding (BBPE). BBPE is more compact than character vocabularies and eliminates out-of-vocabulary (OOV) tokens while remaining more efficient than pure byte models. We demonstrate that contextualizing BBPE embeddings via convolutional or recurrent layers is necessary for optimal performance. Our experiments show that BBPE achieves comparable performance to BPE with only 1/8th the vocabulary size. In multilingual settings, BBPE maximizes vocabulary sharing across languages and enables effective model transfer between languages with non-overlapping character sets.

The Problem: Vocabulary Bloat and Script Isolation

Most modern Neural Machine Translation (NMT) systems rely on subword tokenization, typically via Byte Pair Encoding (BPE) applied to Unicode characters. While effective, this approach has three major drawbacks:

  1. Vocabulary Bloat: Character-rich languages like Chinese or Japanese have thousands of rare characters that consume valuable vocabulary slots.
  2. The OOV Problem: Any character not seen during training becomes an "unknown" token, which can degrade translation quality.
  3. Script Isolation: Models trained on one script (e.g., Latin) cannot easily transfer knowledge to another (e.g., Cyrillic or Sinhala) because their character-based vocabularies do not overlap.

While using raw bytes (0-255) as a vocabulary solves the OOV problem, it results in extremely long sequences that are computationally expensive. BBPE serves as a "middle ground," combining the compactness of subwords with the universality of bytes.

Methodology: BBPE and Contextualization

The core innovation is applying the BPE algorithm to UTF-8 byte sequences instead of Unicode characters. Since UTF-8 encodes characters into 1 to 4 bytes, BBPE can learn subwords that represent partial characters, single characters, or multi-character sequences. This allows the model to represent any possible text using a small, fixed set of byte-based building blocks.

The Necessity of Context

Raw byte-level embeddings are often too fine-grained for a standard Transformer to process efficiently on its own. The authors found that "contextualizing" these embeddings is critical. By inserting either a depth-wise convolutional layer or a bidirectional GRU before the main Transformer layers, the model can better understand the boundaries and meanings of these byte-grams.

Impact of contextualization on BLEU scores across different tokenization schemes.

As shown in the validation results, adding a Bi-GRU contextualization layer provides a significant performance boost (approximately 4% relative BLEU) for fine-grained vocabularies like BBPE and pure bytes.

Multilingual Vocabulary Sharing

In multilingual models, BBPE allows languages with completely different writing systems to share the same underlying byte-level subwords. This maximizes vocabulary efficiency and enables transfer learning across scripts.

Vocabulary sharing across languages for different tokenization methods.

Analysis of the vocabulary demonstrates that a significant portion of BBPE symbols are shared across diverse languages like Arabic, Hebrew, Russian, Korean, and Italian. In contrast, character-based BPE remains largely isolated by script, limiting the potential for cross-lingual knowledge transfer.

Key Results

  • Compactness: On English-German tasks, a BBPE model with a 2K or 4K vocabulary performed as well as a standard BPE model with a 32K vocabulary.
  • Multilingual Superiority: In a 58-language "Many-to-English" task, BBPE models consistently outperformed standard BPE baselines, suggesting that byte-level sharing is a powerful inductive bias for multilingual NMT.
  • Zero-Shot Transfer: The authors successfully transferred a model pretrained on 58 languages to Sinhala, a language with a completely disjoint script. By reusing the byte-level embeddings, the transferred model gained nearly 2 BLEU points over a model trained from scratch.

Conclusion

BBPE offers a robust alternative to character-based subwords for NMT. By operating at the byte level, it eliminates OOV issues and enables massive vocabulary sharing in multilingual contexts. While it requires additional contextualization layers, the resulting models are more compact and versatile, particularly for low-resource or script-diverse translation tasks.

Summary updated

Abstract

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.
BibTeX
@inproceedings{wang2020neural,
  author = {Changhan Wang and Kyunghyun Cho and Jiatao Gu},
  title = {{Neural Machine Translation with Byte-Level Subwords}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=rTfZA6VHpk5},
}
Figure