Abstract
We present a learning architecture for lexical semantic classification problems that supplements task-specific training data with background data encoding general “world knowledge”. The model compiles knowledge contained in a dictionary-ontology into additional training data, and integrates task-specific and background data through a novel hierarchical learning architecture. Experiments on a word sense disambiguation task provide empirical evidence that this “hierarchical classifier” outperforms a state-of-the-art standard “flat” one.
Introduction
Word Sense Disambiguation (WSD) is the task of determining the correct sense of a word in a given context (e.g., chair as furniture vs. chair as a person). A major bottleneck in WSD is data sparsity: standard "flat" classifiers require extensive labeled examples for every specific word sense. However, manually labeling such datasets is labor-intensive, leading to a chronic shortage of training data for rare senses.
This paper proposes addressing this challenge by leveraging "world knowledge"—broad semantic categories that are better represented in existing linguistic resources. While data for a specific sense like "chair" (as a person) might be scarce, data for the general category of "person" is abundant. The challenge lies in systematically integrating this broad knowledge into the specific classification task.
Methodology: Hierarchical Multiclass Perceptron
The authors introduce a Hierarchical Multiclass Perceptron that systematically integrates broad semantic knowledge into the classification task.
1. The Semantic Hierarchy
The model utilizes a two-layer hierarchy derived from WordNet. The bottom layer consists of specific synsets (senses), while the top layer consists of 26 supersenses (e.g., food, person, artifact).
2. Additive Scoring and Learning
To score a potential sense $y$ for an input $x$, the classifier calculates an additive score: $$f(x, y) = \text{Score}(\text{specific sense}) + \text{Score}(\text{broad supersense})$$ This architecture allows the model to "borrow" statistical strength from broad categories when specific sense data is thin. The learning process uses an "ultra-conservative" online algorithm where mistakes trigger weight updates for both the specific sense and its parent supersense.
3. World Knowledge Injection
To populate the broad "supersense" layer, the authors extract thousands of example sentences directly from WordNet. These sentences serve as background training data, providing the model with general contextual knowledge about broad semantic categories.
Experimental Results
The model was evaluated on the Senseval-2 benchmark, focusing on noun data (3,512 training and 1,754 test instances). The hierarchical approach was compared against several baselines, including a flat multiclass perceptron and previous state-of-the-art methods.
| Method | Accuracy (%) |
|---|---|
| Hierarchical Multiclass Perceptron | 71.8 |
| Flat Multiclass Perceptron | 70.4 |
| AdaBoost (previous state-of-the-art) | 69.2 |
| Naive Bayes | 68.0 |
The hierarchical classifier achieved a significant performance boost, reaching 71.8% accuracy. The model was particularly effective on words where the classifiers disagreed, outperforming the flat version 71.5% of the time in those cases. Key insights from the results include:
- Contextual Inheritance: Example sentences for specific concepts act as valid training data for all ancestor categories, effectively expanding the training set.
- Regularization: The hierarchy acts as a regularizer, preventing the model from overfitting to small, specific training sets.
- Data Synergy: The performance gain is only realized when the hierarchy is combined with supplementary "world knowledge" data; without it, the hierarchical model performs similarly to a flat one.
Conclusion and Future Work
This work demonstrates that hierarchical architectures can effectively mitigate data sparsity in WSD by incorporating broad semantic knowledge. Future directions include deepening the hierarchy to exploit WordNet's full depth, implementing non-linear kernels via dual perceptrons, and extending the methodology to verbs and adjectives.
Summary updated