Abstract
This paper introduces Hypernym Class Prediction (HCP), a training strategy designed to mitigate the context sparsity problem in neural language models. By leveraging semantic hierarchies from WordNet, HCP teaches models to predict broad hypernym categories before specific tokens through a curriculum learning approach. Experimental results on WikiText-103 and ARXIV datasets demonstrate that HCP consistently reduces perplexity across various Transformer architectures, particularly improving performance on rare words without increasing inference complexity.
Introduction
Modern neural language models (LMs) excel at capturing complex linguistic patterns but often struggle with the "rare word" problem. Infrequent terms lack sufficient context for the model to learn robust representations, a challenge known as context sparsity. While class-based language models historically addressed this by grouping similar words, successfully integrating this concept into modern Transformers has proven difficult. Previous attempts often sacrificed rare word accuracy or failed to show significant improvements in perplexity. This work bridges that gap by using semantic hierarchies to help models generalize from limited data.
Methodology
The HCP framework consists of two primary components: semantic mapping and curriculum-based training.
Hypernym Mapping
Using the WordNet database, specific nouns are mapped to their corresponding hypernyms. For example, "magnesium" and "iron" are both mapped to the class metallic_element.n.01. This mapping allows the model to treat distinct rare words as members of a shared semantic category, effectively pooling their contextual information during the learning process.
Curriculum Learning
Rather than static multi-task learning, the authors employ a curriculum learning strategy. A "pacing function" determines the probability of substituting a specific token with its hypernym class during training. Early in the process, the model frequently predicts these broader classes, which is a simpler task. As training progresses, the substitution rate is gradually annealed until the model trains exclusively on the original, specific tokens. This transition is crucial; the authors found that simple multi-tasking without a curriculum actually degraded performance.
Experimental Results
The method was evaluated on two major benchmarks: WikiText-103, a large-scale Wikipedia dataset, and ARXIV, a collection of scientific abstracts featuring technical vocabulary and BPE sub-tokens. HCP yielded consistent improvements across all tested architectures, including Transformer-XL and Segatron.
Perplexity Gains and Convergence
On WikiText-103, HCP achieved a 0.6% to 1.9% relative reduction in perplexity. On the ARXIV dataset, gains were even more significant, ranging from 1.3% to 3.1%. While HCP models initially show higher perplexity due to the mixed objective, they eventually surpass standard baselines as the curriculum shifts toward token prediction.
Rare Word Mastery
A key strength of HCP is its ability to improve performance across the entire frequency spectrum. Unlike techniques that trade off common word accuracy for rare word gains, HCP improves both. Pairwise comparisons confirmed that HCP-trained models assign higher probabilities to rare tokens compared to baselines, effectively addressing the context sparsity problem.
Conclusion and Future Work
Hypernym Class Prediction provides a simple yet effective way to incorporate external semantic knowledge into language model training. By using a curriculum to transition from class prediction to token prediction, the model learns more robust representations for rare words without increasing inference time. Future research could expand this approach beyond nouns to other parts of speech and evaluate its impact on downstream tasks like summarization and question answering.
Summary updated