Abstract
Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords, or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese can unnecessarily take up vocabulary slots and limit compactness. Representing text at the level of bytes using the 256-byte set as a vocabulary is a potential solution, but high computational costs have prevented its wide adoption. In this paper, we investigate byte-level subwords, specifically Byte-Level Byte Pair Encoding (BBPE). BBPE is more compact than character vocabularies and eliminates out-of-vocabulary (OOV) tokens while remaining more efficient than pure byte models. We demonstrate that contextualizing BBPE embeddings via convolutional or recurrent layers is necessary for optimal performance. Our experiments show that BBPE achieves comparable performance to BPE with only 1/8th the vocabulary size. In multilingual settings, BBPE maximizes vocabulary sharing across languages and enables effective model transfer between languages with non-overlapping character sets.
The Problem: Vocabulary Bloat and Script Isolation
Most modern Neural Machine Translation (NMT) systems rely on subword tokenization, typically via Byte Pair Encoding (BPE) applied to Unicode characters. While effective, this approach has three major drawbacks:
- Vocabulary Bloat: Character-rich languages like Chinese or Japanese have thousands of rare characters that consume valuable vocabulary slots.
- The OOV Problem: Any character not seen during training becomes an "unknown" token, which can degrade translation quality.
- Script Isolation: Models trained on one script (e.g., Latin) cannot easily transfer knowledge to another (e.g., Cyrillic or Sinhala) because their character-based vocabularies do not overlap.
While using raw bytes (0-255) as a vocabulary solves the OOV problem, it results in extremely long sequences that are computationally expensive. BBPE serves as a "middle ground," combining the compactness of subwords with the universality of bytes.
Methodology: BBPE and Contextualization
The core innovation is applying the BPE algorithm to UTF-8 byte sequences instead of Unicode characters. Since UTF-8 encodes characters into 1 to 4 bytes, BBPE can learn subwords that represent partial characters, single characters, or multi-character sequences. This allows the model to represent any possible text using a small, fixed set of byte-based building blocks.
The Necessity of Context
Raw byte-level embeddings are often too fine-grained for a standard Transformer to process efficiently on its own. The authors found that "contextualizing" these embeddings is critical. By inserting either a depth-wise convolutional layer or a bidirectional GRU before the main Transformer layers, the model can better understand the boundaries and meanings of these byte-grams.
As shown in the validation results, adding a Bi-GRU contextualization layer provides a significant performance boost (approximately 4% relative BLEU) for fine-grained vocabularies like BBPE and pure bytes.
Multilingual Vocabulary Sharing
In multilingual models, BBPE allows languages with completely different writing systems to share the same underlying byte-level subwords. This maximizes vocabulary efficiency and enables transfer learning across scripts.
Analysis of the vocabulary demonstrates that a significant portion of BBPE symbols are shared across diverse languages like Arabic, Hebrew, Russian, Korean, and Italian. In contrast, character-based BPE remains largely isolated by script, limiting the potential for cross-lingual knowledge transfer.
Key Results
- Compactness: On English-German tasks, a BBPE model with a 2K or 4K vocabulary performed as well as a standard BPE model with a 32K vocabulary.
- Multilingual Superiority: In a 58-language "Many-to-English" task, BBPE models consistently outperformed standard BPE baselines, suggesting that byte-level sharing is a powerful inductive bias for multilingual NMT.
- Zero-Shot Transfer: The authors successfully transferred a model pretrained on 58 languages to Sinhala, a language with a completely disjoint script. By reusing the byte-level embeddings, the transferred model gained nearly 2 BLEU points over a model trained from scratch.
Conclusion
BBPE offers a robust alternative to character-based subwords for NMT. By operating at the byte level, it eliminates OOV issues and enables massive vocabulary sharing in multilingual contexts. While it requires additional contextualization layers, the resulting models are more compact and versatile, particularly for low-resource or script-diverse translation tasks.
Summary updated