Introduction
In Natural Language Processing (NLP), the "rare event" problem is a persistent challenge. Due to Zipf’s Law, while a few words appear frequently, the vast majority of words and phrases occur only a handful of times. In typical bilingual corpora, over 60% of word pairs may appear only once. Standard statistical tests, such as the Chi-square ($X^2$) or the log-likelihood-ratio ($G^2$), were originally designed for larger sample sizes, leading many researchers to view them as unreliable for rare data. This paper investigates whether these criticisms are justified and proposes a principled framework for measuring the significance of rare associations.
Methodology
Expected Precision Framework
The paper introduces a method to calculate "expected noise" (the number of pairs expected to occur by chance) and "expected precision" (the probability that an observed association is genuine). By iterating through all possible combinations of word frequencies, the author provides an algorithm to quantify the reliability of any association measure.
Optimized Fisher’s Exact Test
Fisher’s exact test is the mathematical "gold standard" for small sample sizes but is often avoided due to its computational cost. The author demonstrates that by using the gamma function for factorials and recurrence relations for hypergeometric distributions, the test can be optimized for large-scale NLP tasks. In practice, this optimized version only adds a 16% time penalty compared to standard $G^2$ approximations.
LLR vs. P-Value Regression
The author defines $LLR$ as $G^2/2$ and performs a linear regression to evaluate how well this score predicts the negative natural log of the p-value derived from Fisher’s exact test ($-\ln(p)$).
Experimental Setup
The study utilizes a corpus of 500,000 English-French sentence pairs from the Canadian Hansards. This dataset contains approximately 53,000 distinct English words and 66,000 French words, resulting in nearly 20 million word pairs for analysis.
Key Results and Analysis
Predictive Power of LLR
The regression analysis reveals an exceptionally strong relationship between $LLR$ and the significance measured by Fisher’s exact test. With a Pearson’s $r^2$ of 0.999986, $LLR$ serves as a near-perfect predictor of statistical significance. The relationship is approximately defined by the equation: $p\text{-value} \approx e^{-(LLR + 1.15)}$. This suggests that the standard factor of 2 in the $G^2$ formula is primarily an artifact for fitting the Chi-square distribution rather than a necessity for measuring association strength.
Thresholding and Noise Estimation
The data validates common heuristics used in the field. Specifically, an $LLR$ score above 10 is shown to be a highly reliable indicator of a significant association, corresponding to less than 1% expected noise. Interestingly, even for extremely rare events—such as two words appearing only once each but in the same sentence pair—the expected precision remains as high as 95.9%.
Conclusion
The findings demonstrate that $G^2$ (or $LLR$) is a surprisingly robust tool for NLP, even when dealing with rare events. Furthermore, the computational feasibility of Fisher’s exact test—taking only 66.7 minutes to process 20 million pairs on standard hardware—suggests that researchers can and should use exact methods or well-calibrated approximations like $LLR$ rather than relying on arbitrary frequency thresholds. While the current model for expected counts is conservative, it provides a rigorous baseline for future work in word alignment and association discovery.
Summary updated