TL;DR
This paper investigates why a semi-supervised method for automatically generating Word Sense Disambiguation (WSD) training data, based on relevance feedback, only improves performance on biomedical texts. They find that the method's effectiveness is strongly tied to the level of ambiguity in the data, performing best on less ambiguous datasets. This suggests that the inherent ambiguity of language is a critical factor in the success of automated WSD data generation techniques.
What Problem Does This Paper Address?
Supervised Word Sense Disambiguation (WSD) systems generally outperform unsupervised ones, but they require large amounts of manually labeled training data. Creating this data is time-consuming and expensive. To address this, researchers have developed methods to automatically generate labeled examples. One such method, proposed by Stevenson et al. (2008a), uses relevance feedback to expand existing labeled data. While this approach showed promise on biomedical texts, the authors of this paper observed that it didn't perform as well on more general datasets. This paper aims to understand why this method's effectiveness varies across different types of text, hypothesizing that the level of ambiguity in the data is the key factor.
Methods
The core of the paper's methodology involves evaluating Stevenson et al.'s (2008a) relevance feedback approach for generating WSD training examples. This approach works by treating examples of a target sense as "relevant documents" and examples of other senses as "irrelevant documents." Relevance feedback (specifically, Rocchio's algorithm) is then used to generate a set of query terms that identify more relevant documents, which are then used as additional labeled examples.
To analyze the impact of ambiguity, the authors employ two main types of measures:
- Sense Distribution Measures: These include:
- Most Frequent Sense (MFS) baseline: The proportion of examples for a term that use its most common sense. A lower MFS indicates higher ambiguity.
- Number of senses per term: A direct count of the distinct senses a word can take.
- Entropy of the sense distribution: A measure that quantifies the uncertainty or randomness in the distribution of senses. Higher entropy indicates greater ambiguity.
- Semantic Similarity Measures: This novel approach quantifies the similarity in meaning between the possible senses of an ambiguous word. They developed a measure (Equation 1) that computes the average similarity between all possible pairs of senses for a given term. To enable comparison across different sense inventories, they mapped all datasets to WordNet 3.0. They used three specific similarity measures from the WordNet::Similarity package:
- Simple Path: Based on the shortest path between sense nodes in WordNet.
- LCH (Leacock et al.): Also uses shortest path length, combined with depth information.
- JCn (Jiang and Conrath): Uses information theory to assign probabilities to WordNet nodes.
Datasets
The paper evaluates the relevance feedback approach on three datasets:
- NLM-WSD corpus (Weeber et al., 2001): A dataset focused on the biomedical domain. This was the dataset where the original relevance feedback approach showed success.
- Senseval-3 lexical sample task (Mihalcea et al., 2004): A general-domain dataset.
- SemEval-2007 English lexical sample task (Pradhan et al., 2007): Another general-domain dataset, using a coarse-grained version of the task.
For the relevance feedback experiments, the NLM-WSD corpus was subjected to 10-fold cross-validation. The Senseval-3 and SemEval corpora were split into standard training and test sets.
The authors also experimented with the English WaCky corpus for indexing but found it less effective than the British National Corpus (BNC) for retrieving examples.
Key Insights and Ideas
- Ambiguity is King: The central finding is that the effectiveness of the relevance feedback approach for generating WSD training data is highly dependent on the level of ambiguity in the target corpus.
- Biomedical Domain Advantage: The relevance feedback method significantly improved WSD performance on the NLM-WSD corpus (biomedical domain) but not on the general-domain Senseval-3 and SemEval datasets.
- General Domain Challenges: In general domains, the automatically generated examples either worsened performance or provided only marginal improvements when combined with original data, and were often worse than using the original data alone.
- Semantic Similarity as a Predictor: The semantic similarity measure, which quantifies how distinct the meanings of a word's senses are, proved to be a good indicator of how well the relevance feedback approach would perform. Datasets with more semantically distinct senses (lower similarity scores) were less amenable to the approach.
- Mapping Complexity: Mapping different sense inventories to a unified resource like WordNet can be challenging, especially for specialized domains like biomedicine, and might require manual effort and careful handling of unmapped senses.
Key Results
- Performance Drop: When combining original and automatically generated examples, performance dropped by 1.2% for SemEval and 2.9% for Senseval-3 compared to using original data alone.
- NLM-WSD Improvement: For the NLM-WSD corpus, combining original and additional examples yielded a significant improvement (87.9% original, 89.2% combined).
- Ambiguity Ranking: Based on sense distribution measures (number of senses, entropy, MFS), the NLM-WSD corpus was found to be the least ambiguous, Senseval-3 the most ambiguous, and SemEval in between.
- Semantic Similarity Ranking: The semantic similarity analysis corroborated the sense distribution findings, indicating that Senseval-3 had the most ambiguity and NLM-WSD the least.
Limitations and Future Work
- Mapping Limitations: The semantic similarity analysis had to exclude terms with many "None" annotations in the NLM-WSD corpus, reducing the sample size for analysis. It's also noted that "None" annotations might not all refer to the same underlying meaning.
- Sense Inventory Differences: Comparing data sets with different sense inventories is inherently difficult, necessitating mapping to a common resource like WordNet, which itself can be imperfect.
- Future Work: The paper implicitly suggests that future work should focus on developing WSD data generation techniques that are more robust to higher levels of ambiguity. It also highlights the importance of pre-analyzing the ambiguity level of a dataset before applying such methods, using the measures proposed in the paper. The authors suggest that the relevance feedback approach is most appropriate when the level of ambiguity is low.
Summary updated