Abstract
This paper presents a systematic evaluation of the relative contributions of different knowledge sources and machine learning algorithms to the task of Word Sense Disambiguation (WSD). By conducting a multi-dimensional "stress test" using the SENSEVAL-1 and SENSEVAL-2 datasets, the authors demonstrate that a single, well-configured Support Vector Machine (SVM) can outperform the complex ensemble systems of the time. The study highlights the critical interaction between feature selection and algorithm choice, setting new state-of-the-art benchmarks for the field.
Introduction
Natural language is inherently ambiguous, with words like "bank" or "plant" requiring context for correct interpretation. Word Sense Disambiguation (WSD) is the computational task of determining the correct sense of a word in context. By 2002, the field had shifted toward supervised learning using manually tagged corpora. However, research remained fragmented, often focusing on either novel algorithms or novel features in isolation. Furthermore, top-performing systems in competitions like SENSEVAL were often "black boxes"—complex ensembles that made it difficult to identify which components drove performance. This study addresses these gaps through a rigorous, comparative analysis.
Methodology
The researchers evaluated the interaction between two primary factors: linguistic knowledge sources and machine learning algorithms.
Knowledge Sources
Four types of information were extracted from the context surrounding a target word:
- Part-of-Speech (POS): Grammatical categories of the target and its neighbors.
- Surrounding Words: A "bag-of-words" approach capturing all individual words in the nearby context.
- Local Collocations: Specific sequences of words immediately adjacent to the target.
- Syntactic Relations: Identifying the "headword" the target word depends on using a statistical parser.
Learning Algorithms
Four prominent algorithms were compared:
- Support Vector Machines (SVM): Robust to high-dimensional data and noise.
- AdaBoost: An ensemble method combining multiple weak learners.
- Naive Bayes: A probabilistic classifier based on conditional independence assumptions.
- Decision Trees (C4.5): A model that builds classification rules via a tree structure.
Key Insights and Results
The study revealed that the effectiveness of a feature is highly dependent on the algorithm used. For instance, while local collocations were the most beneficial for SVMs, POS tags were more critical for Naive Bayes. Additionally, Naive Bayes required aggressive feature selection to perform well, whereas SVMs proved robust enough to handle all available data without pre-filtering.
Contrary to the prevailing trend of building complex ensembles, the authors found that a single SVM using a linear kernel was sufficient to surpass existing benchmarks.
As shown in the results, the SVM-based approach achieved:
- 65.4% accuracy on SENSEVAL-2, exceeding the best official competition score of 64.2%.
- 79.2% accuracy on SENSEVAL-1, surpassing the top official score of 77.1%.
Paired t-tests confirmed that the SVM approach was statistically significantly better than the top-performing systems of the era, proving that systematic algorithm and feature engineering could outperform "kitchen sink" ensemble strategies.
Conclusion
This work established SVMs as a powerful tool for WSD and emphasized that the choice of learning algorithm is just as vital as the choice of linguistic features. While syntactic relations provided less of a boost than expected, the study's rigorous methodology provided a roadmap for future NLP research. Future work remains in scaling these techniques to "all-words" disambiguation and refining automated feature selection schemes.
Summary updated