Abstract
The effectiveness of a language model is influenced by its token representations, which must encode contextual information and handle the same word form having a plurality of meanings (polysemy). Currently, none of the common language modelling architectures explicitly model polysemy. We propose a language model which not only predicts the next word, but also its sense in context. We argue that this higher prediction granularity may be useful for end tasks such as assistive writing, and allow for more a precise linking of language models with knowledge bases. We find that multi-sense language modelling requires architectures that go beyond standard language models, and here propose a structured prediction framework that decomposes the task into a word followed by a sense prediction task. To aid sense prediction, we utilise a Graph Attention Network, which encodes definitions and example uses of word senses. Overall, we find that multi-sense language modelling is a highly challenging task, and suggest that future work focus on the creation of more annotated training datasets.
Introduction
While standard language models are highly effective at predicting the next word in a sequence, they often struggle to explicitly differentiate between the various meanings of polysemous words. For example, the word "bank" can denote a financial institution or a river's edge depending on the context. While state-of-the-art models like GPT or BERT handle polysemy implicitly through contextual embeddings, they lack an explicit mechanism to identify discrete dictionary senses.
The authors of this paper address this gap by introducing Multi-Sense Language Modelling. This framework aims to predict both the next word and its specific sense. Such explicit tracking is valuable for two primary reasons:
- Knowledge Base Linking: Enabling precise connections to structured data (e.g., distinguishing "bat" the animal from "bat" the sports equipment).
- Assistive Writing: Providing users with context-aware definitions or usage examples for predicted words.
Methodology
The researchers propose a structured prediction framework that breaks the problem into two distinct steps: first, predicting the word; second, predicting its sense.
The Dictionary Graph and Graph Attention Networks
To incorporate external semantic knowledge, a dictionary graph was constructed using WordNet 3.0. This graph consists of "global nodes" for word forms, "sense nodes" for specific meanings, and additional nodes for definitions and example sentences. A Graph Attention Network (GAT) is used to update these node representations, allowing the model to "attend" to dictionary definitions and enrich word embeddings with structured data.
Localized Prediction: The SelectK Approach
The core architecture utilizes a SelectK strategy. A base language model (either a 3-layer GRU or an 8-layer Transformer-XL) identifies the top $K$ most likely next words. A specialized sense-prediction head then chooses the most probable sense from the candidates belonging only to those $K$ words. This localized approach significantly narrows the search space for the sense-prediction task.
Experimental Setup
The models were evaluated on several key datasets:
- SemCor: The primary dataset, featuring tokens manually linked to WordNet senses.
- WikiText-2: Used for pre-training language model components to ensure a robust vocabulary.
- SensEval: An aggregated dataset for additional sense-prediction testing.
- WordNet 3.0: The source for the sense inventory and dictionary graph.
Results and Discussion
The results indicate that multi-sense language modelling is a formidable challenge. Sense prediction is inherently more difficult than word prediction due to the larger "vocabulary" of senses and their "long tail" distribution, where many senses appear infrequently in training data.
Comparative Performance
The Transformer-XL architecture proved superior to the GRU baseline. On the SemCor dataset, the Transformer-XL achieved a sense accuracy of 0.217, whereas the GRU-based model only reached 0.053.
Key Insights
- The SelectK Bottleneck: Sense prediction is heavily dependent on the accuracy of the preceding word prediction. If the correct word is not in the top $K$ candidates, the sense prediction will inevitably fail.
- The Power of K: Optimal results were achieved with $K=1$. Increasing $K$ introduced "distractor" senses, which negatively impacted performance.
- Dictionary Knowledge: While the GAT provided marginal improvements in perplexity, the gains from dictionary definitions were limited, suggesting that current models may not yet fully exploit structured semantic data.
- Ceiling Performance: Using a "Gold LM" (where the correct word is known) resulted in a sense accuracy of 0.90, demonstrating that the sense-prediction logic is effective when the word uncertainty is removed.
Conclusion
The study highlights significant hurdles in multi-sense modelling, particularly data scarcity and the fine-grained nature of WordNet senses. Future research should focus on developing larger sense-annotated datasets and exploring "supersenses"—broader semantic categories—to provide a more manageable level of granularity for practical NLP applications.
Summary updated