TL;DR
This paper investigates how distinguishing between homonymy (unrelated word senses) and polysemy (related word senses) impacts information retrieval (IR) effectiveness. They found that polysemy is more detrimental to IR performance than homonymy, and surprisingly, disambiguation accuracy as low as 55% can still yield benefits when dealing with polysemy.
What Problem Does This Paper Address?
Information retrieval (IR) systems often struggle with lexical ambiguity, where a single word can have multiple meanings (senses). While word sense disambiguation (WSD) has been explored for decades to improve IR, two key questions remain: what level of disambiguation accuracy is needed to see benefits, and how granular should these sense distinctions be? Previous studies often treated all ambiguity similarly, failing to differentiate between homonymy (e.g., "bat" as an animal vs. "bat" as sports equipment) and polysemy (e.g., "mouth" of a river vs. "mouth" of a body). This paper aims to address these gaps by specifically differentiating the impact of homonymy and polysemy on IR and determining the required accuracy for effective disambiguation.
Methods
The authors employ a simulation-based approach using pseudowords to introduce controlled ambiguity into an IR collection.
- Pseudoword Generation: They leverage WordNet, a lexical database, to create pseudowords.
- Homonymy Pseudowords: These are generated by selecting unrelated constituent words, mimicking homonymy.
- Polysemy Pseudowords ("Root Pseudowords"): These are created by selecting constituent words that are related through WordNet's hierarchical structure (hypernyms), thus modeling polysemy. This is a novel approach to generating polysemous pseudowords.
- Pseudo-disambiguation: To simulate WSD, they reduce pseudowords back to their constituent components. This process models disambiguation by assigning a "correct" sense with a certain probability (simulating accuracy) and randomly selecting an incorrect sense otherwise.
- Experimental Setup: They use the WT10G corpus and the TREC 10 Web Track query set. They create three versions of the collection: baseline (unmodified), with added homonymy, and with added polysemy. They then simulate disambiguation at various accuracy levels on the ambiguous collections.
Datasets
- WT10G Corpus: A large IR test collection consisting of 1.69 million documents.
- TREC 10 Web Track Ad-Hoc Query Set: Used for evaluation, consisting of 50 queries.
- WordNet (Version 2.0): Used as a lexical resource for generating pseudowords and understanding semantic relationships.
Key Insights and Ideas
- Polysemy is More Damaging: The paper strongly suggests that polysemy (related senses) is significantly more detrimental to IR performance than homonymy (unrelated senses). This is a crucial distinction often overlooked in prior work.
- Lower Accuracy Threshold for Polysemy: Surprisingly, disambiguation benefits can emerge at much lower accuracy levels (as low as 55%) when dealing with polysemy compared to homonymy.
- Granularity Matters: Differentiating between homonymy and polysemy is essential for understanding the true impact of ambiguity and the effectiveness of disambiguation strategies.
- Coarse-grained vs. Fine-grained: While coarse-grained disambiguation might be more accurate, it may not yield the same IR benefits as fine-grained disambiguation if it fails to capture the nuances of polysemy.
Key Results
- Impact of Ambiguity:
- Adding homonymy resulted in a 2.5% decrease in R-Precision.
- Adding polysemy resulted in a 5.3% decrease in R-Precision, more than double the impact of homonymy.
- Disambiguation Accuracy Thresholds:
- For homonymy, disambiguation accuracy of at least 76% was required to see performance benefits.
- For polysemy, the breakeven point was significantly lower, at 55% accuracy.
- Relative Effects: The study quantifies that adding polysemy has over double the negative impact on retrieval effectiveness compared to adding homonymy.
Limitations and Future Work
- Simulation-based: The study relies on simulations using pseudowords. While this allows for controlled experiments, the authors acknowledge that pseudowords may not perfectly model real-world word usage and ambiguity.
- Uniform Accuracy/Error: The simulation assumes uniform accuracy and error rates across all words and senses. Real-world disambiguation systems often exhibit biases (e.g., performing better on frequent words/senses), which could lead to a more optimistic picture of disambiguation benefits than this uniform simulation suggests.
- Future Work: The authors suggest that more empirical exploration is desirable, but the size of modern IR test collections necessitates continued reliance on simulation. The challenge lies in producing pseudowords that better model real words. They also note that the interaction between homonymy and polysemy in a given query will influence the exact accuracy threshold for performance increases.
Summary updated