Abstract
Shannon entropy is a cornerstone of quantitative linguistics, used to measure everything from grammatical complexity to communicative efficiency. However, estimating entropy from finite corpora is fraught with bias, particularly for the power-law distributions characteristic of human language. In this work, we evaluate several entropy estimators—including Maximum Likelihood (MLE), Miller-Madow, Jackknife, Chao-Shen (CS), and Nemenman-Shafee-Bialek (NSB)—on synthetic and natural language data. Our results show that the standard MLE "plug-in" estimator significantly underestimates entropy and can lead to inflated effect sizes in linguistic studies. We provide a roadmap for more robust estimation, recommending NSB and Chao-Shen as superior alternatives.
Introduction: The Problem of Bias in Entropy Estimation
Linguists frequently employ Shannon entropy to quantify the "uncertainty" or information content of linguistic systems. However, researchers rarely have access to the true probability distribution of a language; instead, they must rely on samples (corpora). This leads to a fundamental statistical challenge: the way we estimate entropy from these samples can systematically skew our conclusions.
Most linguistic studies rely on the Maximum Likelihood Estimator (MLE), or "plug-in" method, which calculates entropy based solely on observed frequencies. Because language follows Zipf’s Law—where a small number of words are common and a vast "long tail" of words are rare—samples inevitably miss many low-frequency items. The MLE assumes that any unobserved word has a probability of zero, which leads to a consistent underestimation of the true entropy. This bias does not simply vanish with more data; it persists, potentially leading researchers to overstate the efficiency or optimality of language.
Methodology and Estimators
The study evaluates six distinct entropy estimators to determine their efficacy in linguistic contexts:
- MLE (Plug-in): The naive baseline that uses observed frequencies.
- Miller-Madow (MM): Applies a correction factor based on the number of possible categories.
- Jackknife (JACK): A resampling technique that attempts to estimate and subtract bias.
- Horvitz-Thompson (HT): Weights observations by their inclusion probability.
- Chao-Shen (CS): An improvement on HT that utilizes "sample coverage" to account for missing classes.
- Nemenman-Shafee-Bialek (NSB): A sophisticated Bayesian approach using a prior designed to be flat over the entropy itself.
The authors tested these estimators using synthetic Zipfian distributions and natural language unigram data from the CELEX database (English, German, Dutch) and Wikipedia (Mongolian, Tagalog).
Key Results: Convergence and Accuracy
The primary finding is that more sophisticated estimators converge to the true entropy much faster than the standard MLE. As shown in Figure 1, the Mean Squared Error (MSE) for estimators like NSB and CS drops significantly faster as the number of samples increases compared to the baseline.
While the "best" estimator can depend on the specific sample size, a general trend emerges: the Chao-Shen (CS) estimator is remarkably robust for extremely small samples, while the NSB estimator becomes the superior choice as the sample size grows, offering the most accurate convergence to the true entropy.
The "Effect Size" Trap in Linguistic Research
Perhaps the most critical insight for the broader linguistic community is the "Effect Size Trap." Poor estimators do not just add random noise; they create systematic false signals. By replicating previous high-profile studies on gender and adjectives using Universal Dependencies (UD) treebanks, the authors demonstrated that using the more accurate NSB estimator can drastically change reported results.
As seen in the table above, the measured effect size (e.g., Normalized Mutual Information) for certain linguistic phenomena was halved when moving from MLE to NSB. For instance, in Italian, the NMI dropped from 20.00% to 11.41%. While the effects remained statistically significant, they were far less dramatic than originally reported using biased estimators.
Conclusion and Recommendations
The paper concludes with a strong recommendation for the field: the use of the MLE "plug-in" estimator for entropy should be discontinued in linguistic research. For unigram distributions, researchers should adopt the NSB estimator whenever possible. In cases of extreme sparsity or specific sub-tasks, the Chao-Shen estimator serves as a robust alternative. By adopting these more sophisticated tools, the field can ensure that its quantitative findings reflect the true underlying structure of human language rather than artifacts of statistical bias.
Summary based on "Estimating the Entropy of Linguistic Distributions" (ACL 2022) by Aryaman Arora, Clara Meister, and Ryan Cotterell.
Summary updated