Better Language Model with Hypernym Class Prediction

He Bai, Tong Wang, Alessandro Sordoni, Peng Shi

Abstract

This paper introduces Hypernym Class Prediction (HCP), a training strategy designed to mitigate the context sparsity problem in neural language models. By leveraging semantic hierarchies from WordNet, HCP teaches models to predict broad hypernym categories before specific tokens through a curriculum learning approach. Experimental results on WikiText-103 and ARXIV datasets demonstrate that HCP consistently reduces perplexity across various Transformer architectures, particularly improving performance on rare words without increasing inference complexity.

Introduction

Modern neural language models (LMs) excel at capturing complex linguistic patterns but often struggle with the "rare word" problem. Infrequent terms lack sufficient context for the model to learn robust representations, a challenge known as context sparsity. While class-based language models historically addressed this by grouping similar words, successfully integrating this concept into modern Transformers has proven difficult. Previous attempts often sacrificed rare word accuracy or failed to show significant improvements in perplexity. This work bridges that gap by using semantic hierarchies to help models generalize from limited data.

Methodology

The HCP framework consists of two primary components: semantic mapping and curriculum-based training.

Hypernym Mapping

Using the WordNet database, specific nouns are mapped to their corresponding hypernyms. For example, "magnesium" and "iron" are both mapped to the class metallic_element.n.01. This mapping allows the model to treat distinct rare words as members of a shared semantic category, effectively pooling their contextual information during the learning process.

WordNet hierarchy paths showing how specific tokens converge on shared hypernym classes.

Curriculum Learning

Rather than static multi-task learning, the authors employ a curriculum learning strategy. A "pacing function" determines the probability of substituting a specific token with its hypernym class during training. Early in the process, the model frequently predicts these broader classes, which is a simpler task. As training progresses, the substitution rate is gradually annealed until the model trains exclusively on the original, specific tokens. This transition is crucial; the authors found that simple multi-tasking without a curriculum actually degraded performance.

Experimental Results

The method was evaluated on two major benchmarks: WikiText-103, a large-scale Wikipedia dataset, and ARXIV, a collection of scientific abstracts featuring technical vocabulary and BPE sub-tokens. HCP yielded consistent improvements across all tested architectures, including Transformer-XL and Segatron.

Perplexity Gains and Convergence

On WikiText-103, HCP achieved a 0.6% to 1.9% relative reduction in perplexity. On the ARXIV dataset, gains were even more significant, ranging from 1.3% to 3.1%. While HCP models initially show higher perplexity due to the mixed objective, they eventually surpass standard baselines as the curriculum shifts toward token prediction.

Validation perplexity curves for small and large Transformer models, showing HCP (blue) outperforming standard baselines (magenta) as training progresses.

Rare Word Mastery

A key strength of HCP is its ability to improve performance across the entire frequency spectrum. Unlike techniques that trade off common word accuracy for rare word gains, HCP improves both. Pairwise comparisons confirmed that HCP-trained models assign higher probabilities to rare tokens compared to baselines, effectively addressing the context sparsity problem.

Conclusion and Future Work

Hypernym Class Prediction provides a simple yet effective way to incorporate external semantic knowledge into language model training. By using a curriculum to transition from class prediction to token prediction, the model learns more robust representations for rare words without increasing inference time. Future research could expand this approach beyond nouns to other parts of speech and evaluate its impact on downstream tasks like summarization and question answering.

Summary updated

Abstract

Class-based language models (LMs) have been long devised to address context sparsity in $n$-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a common WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves perplexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and Arxiv. Our analysis shows that the performance improvement is achieved without sacrificing performance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale.
BibTeX
@inproceedings{bai2022better,
  author = {He Bai and Tong Wang and Alessandro Sordoni and Peng Shi},
  title = {{Better Language Model with Hypernym Class Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.96},
}
Figure