Abstract
Word sense induction (WSI) is a prominent unsupervised approach to lexical disambiguation that clusters word instances based on their contexts. Traditional methods relying on bag-of-words representations often struggle with uninformative or ambiguous contexts. This paper investigates an alternative representation based on lexical substitutes—contextually suitable, meaning-preserving replacements. By utilizing substitutes predicted by a state-of-the-art automatic system and a simple clustering algorithm, the proposed method outperforms traditional context-based representations and competes with complex probabilistic models. Furthermore, results from an oracle based on manually-labeled substitutes suggest significant potential for further improvement as automatic substitution models evolve.
Introduction
Lexical ambiguity is a fundamental challenge in Natural Language Processing. A single word like "paper" can refer to a scholarly article, a news publication, or a physical material depending on its usage. Word Sense Induction (WSI) aims to discover these distinct senses automatically from unlabeled text. While traditional WSI methods rely on the "syntagmatic" context (the words surrounding a target), this paper explores the "paradigmatic" view: what other words could have been used in that exact spot?
Methodology
The core innovation lies in how word instances are represented for clustering. Instead of a simple bag-of-words context (CTX), the authors propose using AUTOLS: a representation based on the embeddings of predicted lexical substitutes.
To generate these substitutes automatically, the researchers employed context2vec, a bidirectional LSTM-based model. These substitutes are then clustered using Affinity Propagation, an algorithm that identifies representative "exemplars" without requiring a pre-specified number of clusters. The authors also tested a late fusion model (AUTOLS+CTX) that combines both substitute and context information.
Key Insights and Ideas
The shift from syntagmatic to paradigmatic representations offers several advantages:
- Noise Filtering: While context words like "reading" might appear with multiple senses of "paper," substitutes like "manuscript" or "pulp" immediately disambiguate the sense.
- Complementarity: The surrounding context and the potential substitutes provide different but complementary information about a word's meaning.
- Simplicity and Scalability: Using a simple clustering algorithm on high-quality substitute embeddings proves more effective than designing intricate probabilistic structures.
As shown in the example above, the system successfully distinguishes between the "scholarly article" sense (Cluster 1), the "newspaper" sense (Cluster 2), and the "material" sense (Cluster 3) of the word "paper."
Results
Evaluations on the SemEval-2010 WSI benchmark demonstrate the superiority of the substitute-based approach. The AUTOLS+CTX model achieved a Supervised Recall (SR) of 0.750, significantly higher than the context-only baseline of 0.693 and the previous state-of-the-art HDP model (0.680).
An "upper bound" experiment using human-provided gold substitutes (GOLDLS) on a 20-word sample showed a dramatic performance leap (Paired F-score increasing from 0.445 to 0.582). This indicates that the WSI task is highly sensitive to the quality of lexical substitutes, and future improvements in automatic substitution will directly translate to better sense induction.
Conclusion
This research highlights the power of lexical substitutes as a robust representation for unsupervised word sense induction. By moving beyond simple context words to contextually appropriate replacements, the authors achieved state-of-the-art results with a relatively simple pipeline. Future work may involve joint modeling of lexical substitution and sense induction to further refine these representations.
Summary updated