Abstract
Word sense induction (WSI), or the task of automatically discovering multiple senses or meanings of a word, has three main challenges: domain adaptability, novel sense detection, and sense granularity flexibility. While current latent variable models are known to solve the first two challenges, they are not flexible to different word sense granularities, which differ very much among words, from aardvark with one sense, to play with over 50 senses. Current models either require hyperparameter tuning or nonparametric induction of the number of senses, which we find both to be ineffective. Thus, we aim to eliminate these requirements and solve the sense granularity problem by proposing AutoSense, a latent variable model based on two observations: (1) senses are represented as a distribution over topics, and (2) senses generate pairings between the target word and its neighboring word. These observations alleviate the problem by (a) throwing garbage senses and (b) additionally inducing fine-grained word senses. Results show great improvements over the state-of-the-art models on popular WSI datasets. We also show that AutoSense is able to learn the appropriate sense granularity of a word. Finally, we apply AutoSense to the unsupervised author name disambiguation task where the sense granularity problem is more evident and show that AutoSense is evidently better than competing models.
Introduction
In Natural Language Processing, Word Sense Induction (WSI) is the task of automatically discovering the various meanings of a word (e.g., "cold" as an illness vs. "cold" as a temperature) without relying on pre-defined dictionaries. While existing models adapt well to new domains, they struggle significantly with sense granularity.
Most models require a human to pre-specify the number of senses ($S$). If $S$ is set too low, nuances are lost; if set too high, models create redundant or nonsensical "garbage" senses. Non-parametric models that attempt to learn the number of senses often fail to do so accurately in practice. AutoSense is designed to handle this variability—from words with a single sense like "aardvark" to those with dozens like "play"—using a single, robust configuration.
Methods: Senses as Topic Distributions and Word Pairings
AutoSense extends the classic Latent Dirichlet Allocation (LDA) framework with two key innovations:
- Senses as Topic Distributions: Instead of treating a "sense" as a single topic, AutoSense introduces a separate latent variable for senses. A sense is represented as a distribution over multiple topics. This mathematical structure allows the model to "throw away" unused senses by assigning zero instances to them, effectively performing automatic garbage collection.
- Target-Neighbor Pairing: Standard models generate words independently. AutoSense forces the model to generate the target word and a neighboring word as a pair $(w_t, w_l)$. This captures specific lexical-semantic relationships (e.g., "cold" paired with "sneeze" vs. "cold" paired with "ice") crucial for distinguishing fine-grained meanings.
The model also employs a switch variable ($x$) to determine if a word should be generated by a specific sense (local context) or a general topic (global context), making it robust to "noisy" words that do not contribute to a word's specific meaning.
Evaluation and Results
The authors evaluated AutoSense on three main benchmarks: SemEval 2010 Task 14, SemEval 2013 Task 13, and Unsupervised Author Name Disambiguation (UAND).
Robustness to Sense Granularity
A striking finding is AutoSense's performance when the number of senses is intentionally overestimated (e.g., setting $S=100$). Unlike competing models whose error rates spike when $S$ is too high, AutoSense maintains low cluster error by ignoring extra slots.
Key Performance Metrics
- Superior Accuracy: On SemEval 2010, AutoSense achieved the highest average score (24.59), outperforming state-of-the-art models like BNP-HC.
- Precision in Sense Counting: AutoSense was remarkably accurate at estimating the number of senses. Its error rate ($\delta$) was only 0.33, whereas competing models like BNP-HC were off by more than 11 senses on average.
- Scalability: In the UAND task, where a single name might belong to over 100 different authors, AutoSense significantly outperformed all baselines, maintaining high performance even as the potential number of "senses" scaled up.
Conclusion
AutoSense successfully addresses the sense granularity problem by treating senses as topic distributions and leveraging target-neighbor word pairings. While the model excels at using text to distinguish meanings, future work could integrate non-textual features—such as co-author networks in disambiguation tasks—to further enhance entity resolution. By bridging the gap between parametric and non-parametric approaches, AutoSense provides a flexible and accurate framework for unsupervised word sense discovery.
Summary updated