TL;DR
This paper proposes a method to estimate the "skewness" (entropy) of word sense distributions automatically. By using this entropy estimate, they can intelligently apply a simple "most frequent sense" heuristic only to words where it's most likely to be accurate, significantly boosting its performance without needing manually labeled data.
What Problem Does This Paper Address?
Word sense disambiguation (WSD) systems aim to figure out which meaning of a word is being used in a given context. A common and surprisingly effective strategy is the "most frequent sense" (MFS) heuristic: just assume the word is being used in its most common sense. This works well because word senses are often "skewed" – one sense is much more frequent than others. However, this heuristic can fail when there isn't enough data to reliably determine the most frequent sense, or when the available data comes from a different domain. This paper tackles the problem of how to automatically identify when the MFS heuristic is likely to be reliable, thus improving its precision without relying on expensive human-annotated data.
Methods
The core of the paper is a method to estimate the entropy of a word's sense distribution. They adapt a technique from McCarthy et al. (2004) that uses a distributional similarity thesaurus (built from the British National Corpus) and WordNet's semantic similarity measure.
Here's a breakdown:
- Distributional Similarity: They first find "neighbor" words that tend to appear in similar contexts as the target word.
- Sense Prevalence Score: For each sense of the target word, they calculate a score based on how similar its neighbors are to that sense. They introduce a modification: the similarity score is weighted by the inverse rank of the neighbor, giving more importance to closer neighbors.
- Probability Distribution: This prevalence score is then normalized across all senses of the word to create a probability distribution over its senses.
- Entropy Estimation: Using this estimated probability distribution, they calculate the entropy. High entropy means senses are more evenly distributed (less skewed), while low entropy means one sense is dominant (highly skewed).
- Exploiting Entropy: The key idea is to use this estimated entropy to decide when to apply the MFS heuristic. They restrict the MFS heuristic to words with an estimated entropy below a certain threshold, assuming these are the words where the MFS is most likely to be correct.
Datasets
- British National Corpus (BNC): Used to build the distributional similarity thesaurus.
- WordNet 1.6: Used for semantic similarity calculations.
- SemCor 1.6: A standard WSD dataset used for evaluation. They focused on polysemous nouns.
- SENSEVAL-2 English All Words (SE2-EAW) dataset: Another hand-tagged test suite used for evaluation.
Key Insights and Ideas
- Entropy as a Confidence Measure: The paper demonstrates that the estimated entropy of a word's sense distribution can serve as a reliable indicator of how confident we can be in the Most Frequent Sense (MFS) heuristic.
- Intelligent Application of MFS: Instead of blindly applying the MFS heuristic to all words, restricting its use to words with low estimated entropy significantly boosts precision. This is a form of "confidence-based" WSD.
- Unsupervised Approach: The method does not require any manually labeled data for training or for estimating the entropy, making it highly practical.
- Modification Matters: Their specific modification to the prevalence score (weighting by inverse rank) showed better results (lower cross-entropy) compared to the original method on SemCor.
Key Results
- On both the SemCor and SE2-EAW datasets, restricting the MFS heuristic to words with low estimated entropy led to significant precision gains.
- For instance, on SemCor, when applying the MFS heuristic only to words with an entropy threshold of 1.0, the precision increased from 68.5% to 85.1% compared to applying it to all words (with an entropy threshold of 0.95).
- The gains over a random baseline were greater at lower entropy levels, suggesting that the method effectively identifies cases where the MFS is genuinely more reliable, not just cases with low polysemy.
- The authors note that their approach is applicable to 9.6% of test data for which there are no training examples, unlike some other methods that rely on parallel training data.
Limitations and Future Work
- Ignoring Local Context: The current method estimates entropy based on general sense distributions and does not incorporate local contextual evidence. The authors suggest coupling the confidence in MFS with contextual evidence as a future direction.
- Coarse-Grained Datasets: They plan to investigate applying their approach to coarse-grained WSD datasets.
- Comparison with Other Methods: They suggest comparing their entropy estimation approach with other methods for estimating sense distributions that do not require hand-labeled data or parallel texts.
Summary updated