TL;DR
This paper introduces a novel method for generating "pseudowords" to evaluate word sense disambiguation (WSD) systems. By leveraging lexical category co-occurrence, their approach creates more realistic ambiguous word pairs than traditional random generation, leading to a more accurate lower bound for WSD performance evaluation.
What Problem Does This Paper Address?
Evaluating word sense disambiguation (WSD) algorithms, especially in new languages or domains, requires sense-tagged corpora, which are expensive and time-consuming to create. A common workaround is using "pseudowords" – artificially created ambiguous words. The standard method involves randomly selecting two words and conflating them (e.g., "banana" + "door" -> "banana-door"). However, this method has drawbacks: the contexts surrounding pseudowords don't reflect real-world usage, leading to overly optimistic performance estimates. Furthermore, it's hard to characterize the types of ambiguity these pseudowords represent. This paper aims to address these limitations by proposing a more principled way to generate pseudowords that better mimic real-world ambiguity.
Methods
The core innovation is using lexical category membership to guide pseudoword generation. Instead of random word selection, the authors analyze the MeSH (Medical Subject Headings) hierarchy to identify pairs of lexical categories that frequently co-occur with ambiguous words in real text. They then select unambiguous words from these identified categories to form pseudowords.
The process involves:
- Identifying ambiguous words: Using the MeSH hierarchy, they truncate codes to group words into broader categories (e.g., "Eye" can fall under "Anatomy" or "Sense Organs").
- Analyzing category co-occurrence: They build a list of frequent two-category pairs that represent ambiguous words in their corpus.
- Generating pseudowords: They sample category pairs based on their observed frequencies and then select random, unambiguous words from those categories to create pseudowords. This ensures the generated pseudowords are derived from categories that are known to exhibit ambiguity in the target domain.
- Controlling for distribution: They also explore variations to generate pseudowords with either more uniform or more skewed underlying word frequency distributions, mimicking real-world scenarios where one sense is often more frequent.
Datasets
- Medline 2003: A collection of 180,226 abstracts was used for training and analysis. The data was split into a training set (2/3, ~120,150 abstracts) and a testing set (1/3, ~60,076 abstracts).
- Abbreviations: A set of 98,841 unique abbreviation-expansion pairs was extracted. These were filtered to retain only those expansions that could be unambiguously mapped to exactly two MeSH categories. This dataset (with 332,020 instances) was used as a benchmark for real, albeit artificial, ambiguity.
Key Insights and Ideas
- Pseudowords as a "Lower Bound": The paper reinforces the idea that pseudowords, when generated appropriately, can provide a more realistic lower bound for WSD performance compared to the standard random method. This is crucial for understanding the true capabilities of WSD systems.
- Category Co-occurrence Matters: The frequency of lexical category co-occurrence is a strong indicator of real-world ambiguity. By mirroring these co-occurrences, pseudowords become more semantically plausible.
- Mimicking Real-World Skew: Real ambiguous words often have a dominant sense. The category-based method allows for generating pseudowords with similar frequency distributions, making the evaluation more challenging and realistic than uniformly distributed senses.
- Abbreviations as a Bridge: The use of abbreviations as an evaluation set is insightful. They represent a middle ground between purely random pseudowords and naturally occurring ambiguous words, validating the hypothesis that category-based pseudowords fall between these two extremes in terms of difficulty.
Key Results
The paper presents accuracy results (in percentages) for different pseudoword generation strategies and context window sizes.
- Category-based pseudowords (Realistic) provide a more accurate lower bound: The "Realistic" pseudowords (category-based, with some frequency control) achieved accuracies that fell between the "Pessimistic" (evenly distributed) and "Optimistic" (standard random) pseudowords. This suggests they are more challenging and thus more informative.
- Abbreviations fall in between: The accuracies for abbreviations (332,020 instances) were also found to fall between the Realistic and Optimistic pseudowords, supporting the idea that category-based pseudowords offer a more realistic evaluation.
- Optimistic pseudowords are indeed too optimistic: The "Optimistic" (standard random) pseudowords consistently yielded the highest accuracies, confirming the authors' initial hypothesis that the standard method overestimates WSD performance. For example, with a context window of 300, Optimistic achieved 81.88% accuracy, while Realistic was around 77.99%.
Limitations and Future Work
- Domain Specificity: The method is demonstrated using the MeSH hierarchy. While the approach is generalizable, its effectiveness might vary across different domains and thesauri.
- Computational Cost: Generating a large, diverse set of pseudowords based on category co-occurrence might be more computationally intensive than simple random selection.
- Future Work: The authors suggest that this method allows for detailed study of specific ambiguity sets by generating many pseudowords from a single category pair. Further exploration of different category granularities and their impact on pseudoword realism is also a promising direction.
Summary updated