Abstract
Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.
Introduction
Standard Neural Machine Translation (NMT) models typically rely on subword tokenization, such as Byte-Pair Encoding (BPE), at the character level. While effective, this approach faces significant challenges when dealing with "character-rich" languages like Chinese or Japanese, or when processing noisy text. In these scenarios, the massive number of unique characters leads to bloated vocabularies where rare characters occupy valuable slots despite being seldom seen during training.
Furthermore, character-based models are inherently script-bound. A model trained on a Latin script (e.g., English) cannot easily adapt to a disjoint script like Sinhala. While raw byte-level vocabularies offer a "universal" solution, they often result in excessively long sequences that are computationally prohibitive. This research proposes Byte-Level BPE (BBPE) as a middle ground, combining the efficiency of subwords with the universality and compactness of bytes.
Methodology: Byte-Level BPE
The core innovation is the shift from characters to bytes as the atomic unit of tokenization. Instead of initiating the BPE merging process with Unicode characters, the process starts with the 256 possible values of a UTF-8 byte.
Tokenization and Contextualization
Text is first converted to UTF-8 byte sequences, and BPE is applied to learn variable-length byte $n$-grams. However, because individual bytes or partial sequences lack the semantic clarity of full characters, the authors introduce a "contextualization" step. By inserting a Depth-Wise Convolutional layer or a Bidirectional GRU (Bi-GRU) immediately after the embedding layer, the model can better learn character boundaries and disambiguate byte sequences.
Decoding
To ensure valid output, a linear-time Dynamic Programming (DP) algorithm is used during inference. This prevents the model from predicting invalid byte sequences that do not form valid Unicode characters, ensuring the output is always recoverable.
Key Results and Insights
The researchers evaluated BBPE across bilingual (En-De, Ja-En, Si-En) and multilingual (58 languages to English) tasks, comparing it against standard Transformer baselines.
Performance and Efficiency
BBPE allows for significantly more compact vocabularies. On English-German (En-De), a BBPE model with a 2K vocabulary achieved a BLEU score of 26.98, nearly matching a standard 32K BPE model (27.31 BLEU). In character-rich contexts like Japanese-English (Ja-En), BBPE 4K+ outperformed the BPE 16K+ baseline in several test sets.
The Importance of Context
A critical finding is that raw byte embeddings are too granular for standard Transformers to handle efficiently. As shown in the experimental results, adding a contextualization layer provides a substantial performance boost (over 4% in BLEU) across various vocabulary sizes.
Universal Script and Transfer Learning
In multilingual settings, BBPE maximizes parameter sharing. Since all languages share the same underlying byte distributions, BBPE enables zero-shot transfer. A model pretrained on 58 languages and fine-tuned on Sinhala-English (a script not seen during pretraining) achieved 9.0 BLEU, significantly higher than the 7.2 BLEU of a model trained from scratch.
Conclusion
BBPE offers a robust alternative to character-level subwords, providing high vocabulary efficiency and enabling cross-script transfer learning. While the resulting sequences are longer than character-BPE, they remain more efficient than pure byte sequences. Future work will focus on optimizing segmentation to further reduce sequence length and exploring broader many-to-many translation settings.
Summary updated