Abstract
Word sense induction (WSI), or the task of automatically discovering multiple senses or meanings of a word, has three main challenges: domain adaptability, novel sense detection, and sense granularity flexibility. While current latent variable models are known to solve the first two challenges, they are not flexible to different word sense granularities, which differ very much among words, from aardvark with one sense, to play with over 50 senses. Current models either require hyperparameter tuning or nonparametric induction of the number of senses, which we find both to be ineffective. Thus, we aim to eliminate these requirements and solve the sense granularity problem by proposing AutoSense, a latent variable model based on two observations: (1) senses are represented as a distribution over topics, and (2) senses generate pairings between the target word and its neighboring word. These observations alleviate the problem by (a) throwing garbage senses and (b) additionally inducing fine-grained word senses. Results show great improvements over the state-of-the-art models on popular WSI datasets. We also show that AutoSense is able to learn the appropriate sense granularity of a word. Finally, we apply AutoSense to the unsupervised author name disambiguation task where the sense granularity problem is more evident and show that AutoSense is evidently better than competing models.
The Challenge of Sense Granularity
The primary goal of Word Sense Induction (WSI) is to automatically discover the various meanings of a word (e.g., "cold" as an illness versus "cold" as a temperature) without relying on a pre-defined dictionary. While existing models like Latent Dirichlet Allocation (LDA) adapt well to new domains, they struggle with sense granularity—the reality that different words have vastly different numbers of meanings.
For instance, a word like "aardvark" has only one sense, while "play" can have over 50. Most current models require users to pre-specify a fixed number of senses. If this number is too low, nuances are lost; if it is too high, the model generates "garbage senses" that do not correspond to real-world meanings. AutoSense was developed to handle any number of senses automatically, effectively pruning unused slots.
Methodology
AutoSense extends the standard LDA framework with two key innovations designed to handle granularity:
- Senses as Topic Distributions: Instead of equating one topic to one sense, AutoSense introduces a separate latent variable for senses. A sense is modeled as a distribution over topics. This allows the model to "throw away" unused senses by assigning them zero probability, effectively pruning the search space.
- Target-Neighbor Pairing: Standard models generate words individually. AutoSense forces the model to generate the target word and a neighboring word as a pair (e.g., "cold" + "sneeze" vs. "cold" + "weather"). This captures lexical semantic features more effectively, helping the model distinguish between very similar fine-grained senses.
The model also utilizes a switch variable ($x$) to determine whether a word in the global context should be generated based on a specific sense or a general background topic. Inference is performed using collapsed Gibbs sampling.
Key Results and Robustness
The authors evaluated AutoSense on the SemEval 2010 and 2013 benchmarks, as well as Unsupervised Author Name Disambiguation (UAND) tasks.
One of the most significant findings is AutoSense's robustness to overestimation. While traditional LDA performance collapses when the initial number of senses ($S$) is set too high (e.g., $S=100$), AutoSense maintains high performance by simply ignoring the extra slots. In experiments with the word "book," the model correctly identified three active senses and left the remaining slots as empty "garbage" senses.
- WSI Performance: On the SemEval 2010 dataset, AutoSense achieved an average score of 24.59, significantly higher than LDA (16.34) and the previous state-of-the-art Hidden Concept (HC) model (22.62).
- Sense Estimation: AutoSense demonstrated the lowest "cluster error" (the difference between predicted and actual sense counts) compared to both parametric and nonparametric models like HDP.
- Author Disambiguation: On the Arnet dataset with $S=100$, AutoSense maintained an F1 score of 58.8, whereas LDA's performance dropped to 8.2. This proves the model's utility in tasks where the number of "senses" (or unique individuals) is unknown and potentially large.
Conclusion
AutoSense effectively addresses the sense granularity problem by decoupling senses from topics and utilizing target-neighbor pairings. While it remains a parametric model requiring an initial maximum $S$, its ability to prune unused senses makes it far more flexible than its predecessors. Future work may involve integrating non-textual features, such as citation networks, to further refine performance in specialized disambiguation tasks.
Summary updated