Training Hybrid Language Models by Marginalizing over Segmentations

Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, Armand JoulinACL 2019 · OpenReview

Introduction

Language modeling typically involves a trade-off between the flexibility of character-level models and the efficiency of word-level models. While word-level models capture semantic meaning effectively, they are plagued by "out-of-vocabulary" (OOV) issues. Character-level models avoid OOV problems but must learn complex dependencies from scratch, often resulting in lower performance. Hybrid models attempt to bridge this gap by predicting both characters and $n$-grams. However, standard hybrid approaches (like Byte Pair Encoding) rely on a single, fixed segmentation of text. This paper proposes a method to marginalize over all possible segmentations, allowing the model to account for the total probability of a string across all its potential decompositions.

The Hybrid Approach

The core of the proposed method is a Transformer-based Hybrid Language Model that predicts a mixture of characters and multi-character $n$-grams.

Marginalization via Dynamic Programming

The primary innovation lies in how the model handles the ambiguity of segmentation. A string like "dog" can be segmented as [d][o][g], [do][g], or [d][og]. Instead of picking one, the authors use dynamic programming to sum the probabilities of all valid paths. This is made computationally feasible by maintaining a character-level context. Because the internal representation ($h_t$) is built solely from the history of individual characters, the context remains invariant regardless of which segmentation path was taken to reach the current position. This decoupling allows for an efficient recurrence relation, similar to the Forward algorithm, to compute the marginal probability of the sequence.

Training and Vocabulary

The model's output layer is expanded to include characters and common $n$-grams (up to length 4) found in the training data. To ensure stability during the initial stages of training, the authors employ a "warm-up" objective where the model learns to predict all valid $n$-grams at every position before transitioning to the full marginalization objective.

Experimental Evaluation

The researchers evaluated the hybrid model on three diverse benchmarks:

  1. Text8: 100M characters of cleaned Wikipedia text.
  2. WikiText2: 10.9M characters of Wikipedia text including numbers and special characters.
  3. Multilingual Wikipedia Corpus (MWC): A dataset spanning seven languages (Czech, German, English, Spanish, Finnish, French, and Russian) to test the model's generalizability.

Results

The proposed hybrid approach consistently outperformed standard character-level Transformer baselines and several state-of-the-art LSTM-based models across all datasets.

Multilingual Performance

On the MWC benchmark, the model demonstrated superior performance across all seven languages, achieving an average of 1.426 bits-per-character (bpc).

Table 2: Performance comparison on the Multilingual Wikipedia Corpus (MWC) across seven languages. The hybrid approach consistently achieves the lowest bits-per-character (bpc).

Benchmarking on Text8

On the Text8 dataset, the model reached 1.156 bpc, significantly improving upon the Transformer baseline and outperforming previous architectures like the BN LSTM and HM LSTM.

Table 1: Results on the Text8 dataset comparing the hybrid approach to previous state-of-the-art models.

Conclusion

By marginalizing over all possible segmentations, this hybrid language model captures the benefits of both character and $n$-gram modeling without the constraints of fixed tokenization. The results across multiple languages and datasets suggest that this approach is a robust improvement for open-vocabulary language modeling. Future work may involve scaling this method to larger Transformer architectures and exploring more extensive $n$-gram vocabularies.

Summary updated

BibTeX
@inproceedings{grave2019training,
  author = {Edouard Grave and Sainbayar Sukhbaatar and Piotr Bojanowski and Armand Joulin},
  title = {{Training Hybrid Language Models by Marginalizing over Segmentations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=XXEKx21eDe},
}
Figure