Introduction
Language modeling typically involves a trade-off between the flexibility of character-level models and the efficiency of word-level models. While word-level models capture semantic meaning effectively, they are plagued by "out-of-vocabulary" (OOV) issues. Character-level models avoid OOV problems but must learn complex dependencies from scratch, often resulting in lower performance. Hybrid models attempt to bridge this gap by predicting both characters and $n$-grams. However, standard hybrid approaches (like Byte Pair Encoding) rely on a single, fixed segmentation of text. This paper proposes a method to marginalize over all possible segmentations, allowing the model to account for the total probability of a string across all its potential decompositions.
The Hybrid Approach
The core of the proposed method is a Transformer-based Hybrid Language Model that predicts a mixture of characters and multi-character $n$-grams.
Marginalization via Dynamic Programming
The primary innovation lies in how the model handles the ambiguity of segmentation. A string like "dog" can be segmented as [d][o][g], [do][g], or [d][og]. Instead of picking one, the authors use dynamic programming to sum the probabilities of all valid paths. This is made computationally feasible by maintaining a character-level context. Because the internal representation ($h_t$) is built solely from the history of individual characters, the context remains invariant regardless of which segmentation path was taken to reach the current position. This decoupling allows for an efficient recurrence relation, similar to the Forward algorithm, to compute the marginal probability of the sequence.
Training and Vocabulary
The model's output layer is expanded to include characters and common $n$-grams (up to length 4) found in the training data. To ensure stability during the initial stages of training, the authors employ a "warm-up" objective where the model learns to predict all valid $n$-grams at every position before transitioning to the full marginalization objective.
Experimental Evaluation
The researchers evaluated the hybrid model on three diverse benchmarks:
- Text8: 100M characters of cleaned Wikipedia text.
- WikiText2: 10.9M characters of Wikipedia text including numbers and special characters.
- Multilingual Wikipedia Corpus (MWC): A dataset spanning seven languages (Czech, German, English, Spanish, Finnish, French, and Russian) to test the model's generalizability.
Results
The proposed hybrid approach consistently outperformed standard character-level Transformer baselines and several state-of-the-art LSTM-based models across all datasets.
Multilingual Performance
On the MWC benchmark, the model demonstrated superior performance across all seven languages, achieving an average of 1.426 bits-per-character (bpc).
Benchmarking on Text8
On the Text8 dataset, the model reached 1.156 bpc, significantly improving upon the Transformer baseline and outperforming previous architectures like the BN LSTM and HM LSTM.
Conclusion
By marginalizing over all possible segmentations, this hybrid language model captures the benefits of both character and $n$-gram modeling without the constraints of fixed tokenization. The results across multiple languages and datasets suggest that this approach is a robust improvement for open-vocabulary language modeling. Future work may involve scaling this method to larger Transformer architectures and exploring more extensive $n$-gram vocabularies.
Summary updated