TL;DR
This paper investigates Latent Semantic Analysis (LSA) for automatically identifying different meanings (senses) of ambiguous words without prior labeling. They found LSA can effectively separate word senses, with cosine distance performing best, though the clusters themselves aren't always "tight."
What Problem Does This Paper Address?
Disambiguating word senses is a fundamental challenge in Natural Language Processing (NLP). Many words have multiple meanings (e.g., "bank" can refer to a financial institution or a river's edge). Traditionally, this requires labeled data or complex rule-based systems. This paper explores whether LSA, a technique for uncovering hidden semantic relationships in text, can automatically group occurrences of an ambiguous word based on their context, thereby discovering distinct word senses in an unsupervised manner.
Methods
The core of the approach involves using LSA to create vector representations (context vectors) for each occurrence of an ambiguous word. These context vectors are derived from the documents in which the word appears. The authors then group these context vectors into clusters, with each cluster ideally representing a distinct word sense.
Key steps include:
- Co-occurrence Matrix Construction: Building a matrix where rows represent word tokens and columns represent contexts (documents).
- LSA and Dimensionality Reduction: Applying Singular Value Decomposition (SVD) to the co-occurrence matrix and then truncating it to a lower-dimensional space. This dimensionality reduction is hypothesized to capture deeper semantic abstractions.
- Clustering: Grouping the LSA-generated context vectors. While the paper evaluates based on pre-assigned sense labels, it aims to mimic unsupervised clustering.
- Evaluation Metrics:
- Sense Discrimination Accuracy: Measuring how well the clusters (represented by their centroids) can correctly identify the sense of a word occurrence from a test set. This is calculated as an upper bound by using pre-assigned sense labels.
- Average Silhouette Value: A measure of how well-separated and cohesive the clusters are. A higher value indicates better-formed clusters.
Datasets
The experiments were conducted on a corpus containing instances of four ambiguous words: "Line," "Hard," "Serve," and "Interest." The corpus is derived from the "line-hard-serve-interest" corpus (Leacock et al., 1993). Specific counts for each word and its senses are provided:
- "Line": 1151 instances, 3 noun senses.
- "Hard": 752 instances, 2 adjective senses.
- "Serve": 1292 instances, 2 verb senses.
- "Interest": 2113 instances, 3 noun senses.
Key Insights and Ideas
- LSA as a Sense Discoverer: The results strongly suggest that LSA can indeed be used to group word occurrences into distinct senses, even without explicit supervision during the clustering process itself (though evaluation uses labels).
- Cosine Distance Reigns Supreme: For most of the tested words, the cosine similarity measure yielded the highest sense discrimination accuracy, making it the preferred metric for this task.
- Dimensionality Reduction's Role: Contrary to some expectations, reducing the dimensionality of the LSA space did not improve sense discrimination accuracy. Instead, accuracy tended to increase monotonically with dimensionality, suggesting that the benefit of dimensionality reduction might lie more in computational efficiency and scalability rather than enhanced semantic discrimination power.
- Separation Over Tightness: While the silhouette values indicated that the sense-based clusters were not particularly "tight" (i.e., points within a cluster weren't extremely close to each other), they were sufficiently "separated" from other clusters to achieve high discrimination accuracy. This implies that the LSA space effectively pushes different senses apart, even if individual occurrences of the same sense are somewhat spread out.
Key Results
- Sense Discrimination Accuracy: Achieved significantly higher accuracy than a baseline (which was the relative percentage of the most frequent sense), demonstrating LSA's utility. For example, accuracy for "Line" reached around 75% at higher dimensions, well above its baseline of 37%.
- Distance Metric Performance: Cosine distance outperformed L1 and L2 for 3 out of 4 words. L1 showed an advantage only for the word "Hard" at low LSA dimensions.
- Silhouette Values: Average silhouette values were generally positive but close to zero, indicating that while clusters were somewhat separated, they weren't exceptionally cohesive.
Limitations and Future Work
- Upper Bound Evaluation: The current evaluation of sense discrimination accuracy provides an upper bound because it relies on pre-established word sense labels. The authors acknowledge that unsupervised clustering algorithms like K-means or EM will likely achieve lower accuracy.
- Context Size: The experiments used the entire document as the context for an ambiguous word. Future work will investigate how varying the context size influences sense discrimination performance.
- Unsupervised Clustering: The next phase of research will involve evaluating clustering obtained without supervision (e.g., using K-means) on the same corpus. The authors anticipate these unsupervised clusters will have better tightness (higher silhouette values) but potentially lower discrimination accuracy compared to the label-based evaluation.
Summary updated