TL;DR
This paper introduces an unsupervised method for distinguishing between different meanings (senses) of a word based solely on its surrounding text. By grouping similar contexts using McQuitty's Similarity Analysis and a rich set of lexical features, the approach aims to overcome the limitations of supervised methods that require expensive sense-tagged data.
What Problem Does This Paper Address?
Word sense discrimination (WSD) is the task of identifying when a word is used with different meanings. While supervised WSD methods achieve high accuracy, they rely on large amounts of manually sense-tagged data, which is time-consuming and expensive to create, limiting their scalability and portability. This paper tackles the challenge of unsupervised WSD, where the goal is to group instances of a word based on their contextual similarity without prior knowledge of the possible senses or any pre-tagged examples. This is distinct from word sense disambiguation (WSD), which typically assigns an instance to one of a predefined set of senses.
Methods
The core of the paper's method is McQuitty's Similarity Analysis, an agglomerative clustering algorithm. The algorithm starts by treating each instance of a target word as its own cluster and iteratively merges the most similar clusters until a predefined number of clusters is reached or the similarity between any pair of clusters falls below a threshold.
The novelty lies in the features used to represent the context of a target word. Instead of relying on simple, localized features, the authors incorporate:
- Unigrams: Individual words appearing in the context.
- Bigrams: Pairs of words that co-occur within a defined window.
- Second-order co-occurrences (SOCs): Words that co-occur with words that co-occur with the target word. This captures more distant relationships.
These features are extracted from a separate "training" corpus to avoid using sense information during feature identification. Instances in the test data are then converted into binary feature vectors. The similarity between these vectors is calculated using either the matching coefficient or the cosine similarity.
Datasets
The authors evaluate their method on two datasets:
- The
linedata: A subset of the English lexical sample data from the SENSEVAL-2 competition. Each instance of the word "line" is manually tagged with one of six possible senses. - SENSEVAL-2 lexical sample data: This dataset contains 73 distinct target words (nouns, verbs, and adjectives) with manually assigned senses.
For both datasets, the authors employ a "training" corpus to extract features and a "test" corpus for evaluation. They also describe preprocessing steps to filter the SENSEVAL-2 data, focusing on the top five most frequent senses and eliminating instances with multiple correct senses.
Key Insights and Ideas
- Rich features are crucial: Using a broader set of features, including second-order co-occurrences, significantly improves the ability to discriminate word senses compared to simpler feature sets.
- Unsupervised methods can be competitive: The unsupervised approach, when using appropriate features and clustering, can achieve precision and recall comparable to or even exceeding the majority sense baseline, especially for certain parts of speech.
- Dataset size matters: The
linedata, with a larger training set, yielded better results than the SENSEVAL-2 data, highlighting the importance of sufficient training data for feature identification. - No single feature/measure reigns supreme: The authors found that different features and similarity measures performed better for different parts of speech, suggesting that feature selection and measure choice are important considerations.
Key Results
The paper presents experimental results in Table 1, showing overall precision and recall for different feature sets and similarity measures across nouns, verbs, adjectives, and the line dataset.
- For nouns, the best performance was achieved with the second-order co-occurrence (soc) features and the cosine similarity measure, yielding a precision of 0.49 and recall of 0.48.
- For verbs, bigrams with the cosine measure performed well, with precision 0.54 and recall 0.45.
- For adjectives, soc features with cosine or matching coefficient achieved the highest precision (0.59) and recall (0.54).
- The
linedataset showed significantly better results than the SENSEVAL-2 data, with soc features and cosine achieving precision 0.25 and recall 0.25.
Crucially, for many individual words within the noun and verb categories, the unsupervised method achieved precision and recall higher than the majority sense baseline. Adjectives were an exception, likely due to their very high frequency majority senses.
Limitations and Future Work
- Limitations:
- The current method's performance on adjectives is limited, possibly due to the high frequency of their majority senses.
- The clustering algorithm tends to produce balanced distributions, which might not accurately reflect skewed sense distributions.
- The evaluation is performed against a "gold standard," but the authors acknowledge that true unsupervised WSD doesn't require pre-existing sense labels.
- Future Work:
- Automatic sense labeling: Develop methods to assign meaningful sense labels to discovered clusters using information from machine-readable dictionaries (e.g., word glosses).
- Feature exploration: Investigate other feature types and variations in feature formulation (e.g., different window sizes, fuzzy matching).
- Similarity measures: Explore additional similarity measures beyond matching coefficient and cosine.
- Data augmentation: Augment SENSEVAL-2 data with training data from the web to improve performance.
Summary updated