Appropriately Incorporating Statistical Significance in PMI

Om P. Damani, Shweta GhongeEMNLP 2013 · DOI

Abstract

Pointwise Mutual Information (PMI) is a fundamental metric for measuring word associations in Natural Language Processing. However, it is notoriously biased toward rare word pairs. While researchers have attempted to mitigate this by incorporating statistical significance, these corrections often fail to improve performance across semantic tasks. This paper identifies a mathematical flaw in the definition of "random trials" within existing significance-weighted PMI variants. By proposing a "max-frequency" correction—using the more frequent word as the statistical anchor—the authors introduce PMIs and sPMId. These revised measures consistently outperform standard PMI and other popular co-occurrence metrics on benchmarks including WordSim-353 and TOEFL synonym selection.

Introduction: The Rare Word Problem in PMI

In Natural Language Processing, determining the strength of association between words (e.g., "doctor" and "nurse") is critical for tasks ranging from information retrieval to machine translation. Pointwise Mutual Information (PMI) has long been the standard tool for this purpose.

Despite its popularity, PMI suffers from a significant "rare word" problem: it tends to assign disproportionately high scores to word pairs that appear together very infrequently simply because their individual probabilities are low. To address this, various "significance-aware" versions of PMI have been proposed, aiming to weight scores by the amount of available evidence. Surprisingly, these corrections often perform worse than the original PMI. This paper investigates why these corrections fail and proposes a mathematically sound alternative.

Methods: Redefining the "Random Trial"

The authors pinpoint the failure of previous significance-weighted measures to a misunderstanding of what constitutes a "random trial" for word co-occurrence. When calculating the significance of a pair $(x, y)$, one must define the number of opportunities they had to appear together.

The "Old Way" vs. The "New Way"

Previous methods, such as $PMI_{sig}$, assumed that the evidence for a pair is proportional to the minimum frequency of the two words ($\min(f(x), f(y))$). The reasoning was that a pair cannot occur more often than its rarest member.

The authors argue that this is incorrect. Instead, the maximum frequency ($\max(f(x), f(y))$) should be used. Every time the more frequent word appears in the corpus, it creates a "site" or a "trial" where the less frequent word could have appeared. Using the maximum frequency as the anchor provides a more accurate statistical bound for the expected co-occurrence.

Comparison of original and revised formulas for significance-weighted PMI.

Based on this insight, two revised measures were introduced:

  1. PMIs: A revision of $PMI_{sig}$ that weights the PMI value by the square root of the maximum frequency.
  2. sPMId: A document-count based measure (revising $cPMId$) that uses the maximum document frequency as the anchor.

Evaluation and Datasets

The revised measures were evaluated using a 1.24 billion-word Wikipedia corpus across four standard benchmarks:

  • WordSim-353: Measures semantic relatedness.
  • Li et al. (2006): Evaluates sentence similarity across 65 pairs.
  • ESL (English as a Second Language): 50 synonym selection questions.
  • TOEFL: 80 synonym selection questions.

Key Results

The revised measures, PMIs and sPMId, demonstrated superior consistency and performance compared to both standard PMI and previous significance-weighted attempts.

Performance comparison of various co-occurrence measures across semantic benchmarks.

As shown in the results table, the "max-frequency" correction consistently outperformed basic PMI. Notably, on the TOEFL synonym task, both sPMId and PMIs achieved a score of 0.61, surpassing standard PMI (0.59) and the original $PMI_{sig}$ (0.56). Furthermore, these relatively simple, low-cost measures proved competitive with more complex, resource-intensive methods like Latent Semantic Analysis (LSA) and various knowledge-based approaches.

Conclusion

The primary contribution of this work is the realization that statistical significance in word association depends on the number of opportunities for co-occurrence, which is best represented by the frequency of the more common word in a pair. By making a simple adjustment—switching a min function to a max function—the authors significantly improved the robustness of PMI-based measures. This demonstrates that mathematically sound "low-cost" co-occurrence measures remain highly relevant and effective even in the era of complex distributional models.

Summary updated

BibTeX
@inproceedings{damani2013appropriately,
  author = {Om P. Damani and Shweta Ghonge},
  title = {{Appropriately Incorporating Statistical Significance in PMI}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1017},
}
Figure