Multi-Sense Language Modelling

Andrea Lekkas, Peter Schneider-Kamp, Isabelle Augenstein

Abstract

The effectiveness of a language model is influenced by its token representations, which must encode contextual information and handle the same word form having a plurality of meanings (polysemy). Currently, none of the common language modelling architectures explicitly model polysemy. We propose a language model which not only predicts the next word, but also its sense in context. We argue that this higher prediction granularity may be useful for end tasks such as assistive writing, and allow for more a precise linking of language models with knowledge bases. We find that multi-sense language modelling requires architectures that go beyond standard language models, and here propose a structured prediction framework that decomposes the task into a word followed by a sense prediction task. To aid sense prediction, we utilise a Graph Attention Network, which encodes definitions and example uses of word senses. Overall, we find that multi-sense language modelling is a highly challenging task, and suggest that future work focus on the creation of more annotated training datasets.

Introduction

Standard language models (LMs) typically treat words as atomic units, mapping each token to a single vector representation. However, natural language is rife with polysemy, where a single word form (e.g., "bank") can represent vastly different concepts depending on context—from a financial institution to the side of a river. While modern contextualized embeddings capture these nuances implicitly, they lack explicit grounding in discrete dictionary senses. This limitation hinders the integration of LMs with structured knowledge bases like WordNet or Wikipedia.

Methodology

The authors propose a "localized prediction" framework that decomposes the multi-sense task into two sequential steps.

Two-Stage Prediction Framework

First, a base language model (either a 3-layer GRU or an 8-layer Transformer-XL) generates a probability distribution over the word vocabulary. Second, the model predicts the specific sense of the predicted word. To manage the vast sense vocabulary, a "SelectK" mechanism restricts the sense prediction to only those senses belonging to the top $K$ most likely words from the first stage.

The multi-sense language modelling architecture, showing the transition from word prediction to sense prediction via a dictionary graph.

Structured Knowledge Integration

To support sense disambiguation, the model incorporates a Dictionary Graph derived from WordNet 3.0. This graph contains "global nodes" for words, "sense nodes" for specific meanings, and auxiliary nodes for definitions and example sentences.

A portion of the dictionary graph for the word 'bank', illustrating the connections between the word, its various senses, and their associated definitions and examples.

A Graph Attention Network (GAT) is employed to update these node representations. By aggregating information from definitions and semantic neighbors (synonyms/antonyms), the GAT provides the model with a richer, knowledge-grounded representation of each candidate sense before the final softmax prediction.

Evaluation and Results

The framework was evaluated primarily on SemCor, a subset of the Brown Corpus annotated with WordNet senses.

  • The Granularity Gap: Predicting the correct sense proved significantly harder than predicting the next word. On SemCor, a Transformer-XL achieved 24.1% word accuracy but only 21.7% sense accuracy.
  • The SelectK Bottleneck: Sense prediction is highly sensitive to the underlying LM's performance. If the correct word is not within the top $K$ candidates, sense prediction fails by default.
  • Gold LM Performance: When provided with the "gold" (correct) next word, the architecture's disambiguation accuracy reached 90%, demonstrating that the sense-prediction component is effective when the word-level bottleneck is removed.
  • Baseline Comparisons: The "Most Frequent Sense" (MFS) heuristic remains a formidable baseline, often outperforming complex neural similarity measures due to the extreme data sparsity in sense-annotated corpora.

Discussion and Future Work

Multi-sense language modelling is constrained by a severe lack of annotated data; the UFSAC contains only 2.0M sense-tagged words, a fraction of the data used for standard LM training. Furthermore, WordNet's fine-grained senses may be too specific for current architectures. Future research should explore "supersenses"—broader semantic categories—and more sophisticated encoders for dictionary definitions to better bridge the gap between static knowledge and dynamic context.

Summary updated

Abstract

The effectiveness of a language model is influenced by its token representations, which must encode contextual information and handle the same word form having a plurality of meanings (polysemy). Currently, none of the common language modelling architectures explicitly model polysemy. We propose a language model which not only predicts the next word, but also its sense in context. We argue that this higher prediction granularity may be useful for end tasks such as assistive writing, and allow for more a precise linking of language models with knowledge bases. We find that multi-sense language modelling requires architectures that go beyond standard language models, and here propose a structured prediction framework that decomposes the task into a word followed by a sense prediction task. To aid sense prediction, we utilise a Graph Attention Network, which encodes definitions and example uses of word senses. Overall, we find that multi-sense language modelling is a highly challenging task, and suggest that future work focus on the creation of more annotated training datasets.
BibTeX
@misc{lekkas2020multi,
  author = {Andrea Lekkas and Peter Schneider-Kamp and Isabelle Augenstein},
  title = {{Multi-Sense Language Modelling}},
  year = {2020},
  eprint = {2012.05776},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.05776},
}
Figure