Abstract
The effectiveness of a language model is influenced by its token representations, which must encode contextual information and handle the same word form having a plurality of meanings (polysemy). Currently, none of the common language modelling architectures explicitly model polysemy. We propose a language model which not only predicts the next word, but also its sense in context. We argue that this higher prediction granularity may be useful for end tasks such as assistive writing, and allow for more a precise linking of language models with knowledge bases. We find that multi-sense language modelling requires architectures that go beyond standard language models, and here propose a structured prediction framework that decomposes the task into a word followed by a sense prediction task. To aid sense prediction, we utilise a Graph Attention Network, which encodes definitions and example uses of word senses. Overall, we find that multi-sense language modelling is a highly challenging task, and suggest that future work focus on the creation of more annotated training datasets.
Introduction
Standard language models (LMs) typically treat words as atomic units, mapping each token to a single vector representation. However, natural language is rife with polysemy, where a single word form (e.g., "bank") can represent vastly different concepts depending on context—from a financial institution to the side of a river. While modern contextualized embeddings capture these nuances implicitly, they lack explicit grounding in discrete dictionary senses. This limitation hinders the integration of LMs with structured knowledge bases like WordNet or Wikipedia.
Methodology
The authors propose a "localized prediction" framework that decomposes the multi-sense task into two sequential steps.
Two-Stage Prediction Framework
First, a base language model (either a 3-layer GRU or an 8-layer Transformer-XL) generates a probability distribution over the word vocabulary. Second, the model predicts the specific sense of the predicted word. To manage the vast sense vocabulary, a "SelectK" mechanism restricts the sense prediction to only those senses belonging to the top $K$ most likely words from the first stage.
Structured Knowledge Integration
To support sense disambiguation, the model incorporates a Dictionary Graph derived from WordNet 3.0. This graph contains "global nodes" for words, "sense nodes" for specific meanings, and auxiliary nodes for definitions and example sentences.
A Graph Attention Network (GAT) is employed to update these node representations. By aggregating information from definitions and semantic neighbors (synonyms/antonyms), the GAT provides the model with a richer, knowledge-grounded representation of each candidate sense before the final softmax prediction.
Evaluation and Results
The framework was evaluated primarily on SemCor, a subset of the Brown Corpus annotated with WordNet senses.
- The Granularity Gap: Predicting the correct sense proved significantly harder than predicting the next word. On SemCor, a Transformer-XL achieved 24.1% word accuracy but only 21.7% sense accuracy.
- The SelectK Bottleneck: Sense prediction is highly sensitive to the underlying LM's performance. If the correct word is not within the top $K$ candidates, sense prediction fails by default.
- Gold LM Performance: When provided with the "gold" (correct) next word, the architecture's disambiguation accuracy reached 90%, demonstrating that the sense-prediction component is effective when the word-level bottleneck is removed.
- Baseline Comparisons: The "Most Frequent Sense" (MFS) heuristic remains a formidable baseline, often outperforming complex neural similarity measures due to the extreme data sparsity in sense-annotated corpora.
Discussion and Future Work
Multi-sense language modelling is constrained by a severe lack of annotated data; the UFSAC contains only 2.0M sense-tagged words, a fraction of the data used for standard LM training. Furthermore, WordNet's fine-grained senses may be too specific for current architectures. Future research should explore "supersenses"—broader semantic categories—and more sophisticated encoders for dictionary definitions to better bridge the gap between static knowledge and dynamic context.
Summary updated