Characterising Measures of Lexical Distributional Similarity

Julie Weeds, David Weir, Diana McCarthyCOLING 2004

TL;DR

This paper investigates why different computational methods for measuring word similarity produce vastly different sets of "neighboring" words. They reveal that these differences are largely driven by how each method handles word frequency, leading to distinct biases towards high- or low-frequency words. This understanding is crucial for selecting the right similarity measure for specific NLP tasks, such as determining if a multi-word expression's meaning is predictable from its parts.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), understanding the relationships between words is fundamental. "Distributional similarity" is a key concept, suggesting that words appearing in similar contexts likely share semantic similarities. This idea is leveraged in various applications, from building better language models to automatically generating thesauruses. However, there's a bewildering array of computational methods to measure this distributional similarity, and it's unclear which one is "best." Previous work often compared these measures against manually created semantic resources or evaluated them on specific tasks, but this paper argues for a deeper understanding: analyzing why these measures differ by examining the statistical and linguistic properties of the neighbor sets they produce. This allows for predicting which measure might be most suitable for a given task before extensive experimentation.

Methods

The authors analyze ten different distributional similarity measures, including well-known ones like cosine similarity, Jensen-Shannon divergence, and measures based on Pointwise Mutual Information (PMI). They focus on how these measures rank words based on their co-occurrence patterns in text. A key aspect of their analysis is examining the "overlap" between the sets of nearest neighbors generated by different measures for the same target word. They also introduce a metric called "bias" to quantify how strongly a measure favors high-frequency or low-frequency words as neighbors, relative to the target word's frequency.

Datasets

The primary dataset used is the British National Corpus (BNC). They extract direct-object dependency relations for 2000 nouns. This set is further divided into high-frequency nouns (frequency > 500) and low-frequency nouns (ranked 3001-4000) to analyze frequency-related biases. For the hyponymy analysis, they use pairs from WordNet 1.6. For the compositionality task, they utilize data from McCarthy et al. (2003) concerning phrasal verbs and their simplex constituents.

Key Insights and Ideas

  • Frequency is King: The most significant factor differentiating neighbor sets from various similarity measures is their bias towards high or low-frequency words.
  • Three Classes of Measures: The measures can be broadly categorized:
  • High-frequency seekers: Measures like cosine (cm), Jensen-Shannon (js), alpha-skew (α), and cosine probability (cp) tend to select high-frequency neighbors regardless of the target word's frequency.
  • Low-frequency seekers: The precision measure (P) strongly favors low-frequency neighbors.
  • Frequency-matching: Measures like Jaccard (ja), Jaccard+MI (ja+mi), Lin's measure (lin), and harmonic mean (hm) tend to select neighbors with similar frequencies to the target word.
  • Distributional Generality and Hyponymy Link: Measures that exhibit high "recall" (finding many relevant co-occurrences) tend to select distributionally more general words, which correlates with hyponymy (e.g., "dog" is a hyponym of "animal"). Conversely, high "precision" measures (finding highly relevant but fewer co-occurrences) align with hypernymy.
  • Frequency, Generality, and Semantics Intertwined: There's a three-way connection: word frequency, distributional generality (measured by recall/precision), and semantic generality (hyponymy/hypernymy) are linked. Specifically, hypernyms are often more frequent than their hyponyms.
  • Compositionality Matters: The choice of similarity measure significantly impacts the success of tasks like determining the compositionality of collocations (e.g., whether "strong tea" means more than just "tea"). Measures that capture distributional generality (high recall) perform better here.

Key Results

  • Significant Variation: Even for many pairs of measures, the overlap in their top 200 nearest neighbors was not as high as one might expect from random chance, indicating substantial differences.
  • Frequency Bias Quantified: Table 4 shows clear biases. For instance, the cosine measure (cm) has a biashigh score of 0.90 for high-frequency target nouns and 0.87 for low-frequency ones, indicating a strong preference for high-frequency neighbors. Precision (P) shows the opposite, with scores of 0.12 and 0.04, respectively.
  • Hyponymy Prediction: The direction of hyponymy relations was predicted correctly by precision-recall values in 71% of cases and by relative frequency in 70% of cases.
  • Compositionality Correlation: Using the recall measure, they found a significant correlation (0.160, p=0.0475) between similarity scores and human judgments of phrasal verb compositionality, suggesting recall-based measures are better suited for this task. Precision showed a negative correlation.

Limitations and Future Work

  • Limited Scope: The analysis primarily focused on nouns and direct-object relationships, with some extension to verbs. Further work could explore other grammatical relations and word classes.
  • Future Directions: The authors suggest using these findings to organize distributional similarity measures into hierarchies. They also propose exploring the impact of frequency characteristics in other NLP applications and identifying further characteristics to classify similarity measures. The observed three-way correlation between frequency, distributional generality, and semantic generality is a rich area for further investigation.

Summary updated

BibTeX
@inproceedings{weeds2004characterising,
  author = {Julie Weeds and David Weir and Diana McCarthy},
  title = {{Characterising Measures of Lexical Distributional Similarity}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2004},
}
Figure