Abstract
Class-based language models (LMs) have been long devised to address context sparsity in $n$-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a common WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves perplexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and Arxiv. Our analysis shows that the performance improvement is achieved without sacrificing performance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale.
Introduction
Modern neural language models (LMs) excel at capturing linguistic patterns, but they still struggle with the "long tail" of language: rare words. Because rare words appear in very few contexts, the model has difficulty learning robust representations for them—a challenge known as context sparsity.
While older, $n$-gram-based models used "class-based" approaches to group similar words together to pool their statistical strength, successfully porting this idea to modern neural Transformers has been difficult. Previous attempts often focused on specific tasks like word-sense disambiguation rather than improving core generative quality. This research bridges that gap by using hierarchical word relationships to help the model "generalize" before it "specializes."
Hypernym Class Prediction (HCP)
The proposed training enrichment, Hypernym Class Prediction (HCP), leverages the WordNet lexical database to map specific nouns to their "hypernyms" (broader categories). For example, "magnesium" and "iron" both map to the hypernym "metallic_element.n.01."
The training process follows a curriculum learning approach:
- Mapping: Rare nouns (determined by a frequency threshold) are mapped to a hypernym at a specific hierarchy depth.
- Substitution: During early training, a subset of tokens is replaced with these hypernym labels, forcing the model to predict the category instead of the specific word.
- Annealing: As training progresses, a "pacing function" gradually reduces the probability of substitution, eventually switching entirely to standard token prediction.
- Mixed Vocabulary: The model uses a vocabulary including both standard tokens and hypernym classes only during the training phase.
Crucially, this is a training-only modification. At inference time, the model is a standard Transformer that predicts regular tokens, meaning there is zero overhead or requirement for WordNet during deployment.
Experimental Results
The authors evaluated HCP on WikiText-103 and Arxiv across various architectures, including standard Transformers, Transformer-XL, and Segatron. HCP consistently outperformed baseline models, yielding a 0.6–1.9% relative reduction in perplexity on WikiText-103 and a 1.3–3.1% reduction on Arxiv.
Rare Word Robustness
A key insight is Implicit Context Aggregation: by forcing the model to predict a shared hypernym for different words early on, the model learns that the surrounding contexts are similar. This helps build better representations for rare words that would otherwise be seen too infrequently to learn effectively.
As shown in the analysis across frequency strata, HCP improves performance across the board. Unlike many techniques that improve rare word performance at the cost of frequent word accuracy, HCP improved performance across all frequency buckets, including the rarest words (frequency < 20).
Discussion and Future Work
The researchers found that the "easy-to-hard" transition provided by curriculum learning was essential; multi-task learning (predicting tokens and hypernyms simultaneously) actually degraded performance. Additionally, the hierarchy depth matters; Depth 6 in the WordNet tree was found to be the "sweet spot" for performance.
Current limitations include a focus on nouns, as WordNet’s noun hierarchy is the most robust. Extending this to verbs or adjectives remains an open challenge. Furthermore, the approach relies on WordNet, which is most mature for English. Future work will investigate whether these perplexity improvements translate to better performance on downstream tasks like summarization or question answering.
Summary updated