Frequency Estimates for Statistical Word Similarity Measures

Egidio L. Terra, Charles L. A. ClarkeNAACL 2003

TL;DR

This paper investigates how different methods for estimating word co-occurrence frequencies impact the performance of statistical word similarity measures. They find that document-retrieval-based Pointwise Mutual Information (DR-PMI) with a specific window size performs best across various language understanding tasks, outperforming previous benchmarks.

What Problem Does This Paper Address?

Measuring word similarity is crucial for many Natural Language Processing (NLP) tasks, including language modeling and information retrieval. A core component of these similarity measures is estimating how often words appear together (co-occurrence frequencies). However, there are many ways to define and estimate these frequencies, such as using fixed-size windows of text or considering entire documents. The authors aim to systematically study how these different estimation methods, along with varying corpus sizes, affect the accuracy of word similarity measures on challenging language tasks, specifically synonym identification.

Methods

The paper compares various statistical word similarity measures, including Pointwise Mutual Information (PMI), Chi-Squared (CHI), Average Mutual Information (MI), and Likelihood Ratio (LL). They also explore context-aware variants like Cosine of PMI (CP), L1 norm, Contextual Average Mutual Information (AMIC), and Contextual Jensen-Shannon Divergence.

The key novelty lies in their comparative study of two primary co-occurrence frequency estimation approaches:

  • Window-oriented: Counts co-occurrences within a defined sliding window of words. They explore various window sizes (from 2 to 256 words).
  • Document-oriented: Considers words co-occurring within the same document. They also explore variants that use a fixed-size window within documents.

They also investigate the impact of corpus size on these estimates.

Datasets

The experiments were conducted on three test sets:

  • TOEFL test set: 80 synonym questions, originally used by Landauer and Dumais (1997) and Turney (2001).
  • TS1: 50 synonym practice questions, used by Turney (2001).
  • TS2: 60 synonym practice questions, extracted from a TOEFL practice guide.

The corpus used for frequency estimation was a terabyte of web data crawled in 2001, containing approximately 53 billion words and 77 million documents.

Key Insights and Ideas

  • DR-PMI shines: The document-retrieval approach to estimating PMI, particularly with a window size of 16-32 words, consistently yielded the best results across all test sets, especially when context was not provided.
  • Context isn't always king: Surprisingly, for some test sets (TS1 and TS2), context-aware similarity measures did not significantly outperform their non-contextual counterparts, and in some cases, performed worse. This suggests that either the context provided was not informative enough, or the methods struggled to effectively utilize it.
  • Window size matters: The optimal window size for frequency estimation varied, but for DR-PMI, sizes in the mid-range (16-32 words) proved most effective.
  • Corpus size has diminishing returns: While a large corpus is essential, the impact of increasing corpus size beyond a certain point (around 300-500 GBytes for some measures) showed diminishing returns, suggesting an asymptote in performance gains.
  • MI and LL are similar: The Average Mutual Information (MI) and Log-Likelihood (LL) measures produced identical results, indicating that the binomial distribution assumption for word occurrences is a reasonable approximation.

Key Results

  • The best performing combination (DR-PMI with a window size of 16-32 words) achieved 81.25% accuracy on the TOEFL test set, a significant improvement over previous reported results of 73.75% (Turney, 2001) and 64.5% (Landauer and Dumais, 1997).
  • On TS1 (no context), their best result was 72%, outperforming Turney's 66%.
  • On TS2 (no context), the best performance reached 75%.
  • Overall, without any parameter tuning, the methods achieved at least 75% accuracy on all test sets.

Limitations and Future Work

  • Corpus differences: The authors acknowledge that direct comparison with previous work using Altavista data is difficult due to unknown differences in corpus construction and query evaluation.
  • Contextual limitations: The paper highlights that contextual measures did not always improve performance, suggesting that further research into more effective context utilization is needed.
  • Multi-word issues: Handling multi-word expressions in target words or alternatives presented challenges, requiring ad-hoc adjustments.
  • Future work: The authors suggest investigating the impact of corpus size more thoroughly and exploring alternative methods for context utilization. They also note that the syntax-based approach, while promising, was impractical for their specific language tests.

Summary updated

BibTeX
@inproceedings{terra2003frequency,
  author = {Egidio L. Terra and Charles L. A. Clarke},
  title = {{Frequency Estimates for Statistical Word Similarity Measures}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}

Loading related content...

Figure