Estimating the Entropy of Linguistic Distributions

Aryaman Arora, Clara Meister, Ryan Cotterell

Abstract

Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropy must typically be estimated from observed data because researchers do not have access to the underlying probability distribution that gives rise to these data. While entropy estimation is a well-studied problem in other fields, there is not yet a comprehensive exploration of the efficacy of entropy estimators for use with linguistic data. In this work, we fill this void, studying the empirical effectiveness of different entropy estimators for linguistic distributions. In a replication of two recent information-theoretic linguistic studies, we find evidence that the reported effect size is over-estimated due to over-reliance on poor entropy estimators. Finally, we end our paper with concrete recommendations for entropy estimation depending on distribution type and data availability.

Introduction

Entropy is a cornerstone of information theory, serving as a vital metric for linguists to quantify the complexity of morphological systems, the optimality of lexicons, and the efficiency of communication. However, a fundamental challenge persists: researchers never have access to the "true" probability distribution of a language. Instead, they must rely on finite samples—corpora—to estimate these values.

The standard approach in the field has long been plug-in estimation (Maximum Likelihood Estimation, or MLE). While computationally simple, this method is mathematically guaranteed to underestimate true entropy because it fails to account for "unseen" events—words or structures that exist in the language but do not appear in a specific sample. Given that linguistic data typically follows Zipf’s Law, where a few items are highly frequent and a vast "long tail" of items is extremely rare, this bias is particularly aggressive and problematic for linguistic research.

Comparative Methodology

To address this gap, this study provides the first comprehensive empirical comparison of six entropy estimators specifically within the context of linguistic data:

  1. MLE (Plug-in): The naive baseline that calculates entropy directly from observed frequencies.
  2. Miller-Madow (MM): A correction method that adjusts the MLE based on the size of the "alphabet" or support.
  3. Jackknife (JACK): A resampling technique that estimates bias by systematically omitting data points.
  4. Horvitz-Thompson (HT): A method that weights observations by their inclusion probability to better account for rare items.
  5. Chao-Shen (CS): A sophisticated approach combining HT with "sample coverage" (derived from Good-Turing smoothing) to estimate the probability of missing classes.
  6. Nemenman-Shafee-Bialek (NSB): A Bayesian estimator that utilizes a hyperprior to ensure the prior distribution over entropy itself is uniform, preventing the estimator from being dominated by its own hyperparameters in low-data regimes.

The estimators were evaluated using Mean Squared Error (MSE) and Mean Absolute Bias (MAB) across synthetic distributions (Symmetric Dirichlet and Zipfian) and natural language data (CELEX database and Wikipedia dumps for English, German, Dutch, Mongolian, and Tagalog).

The Impact of Estimation Bias

The study reveals that the "plug-in trap" is a significant issue in contemporary linguistics. The MLE estimator consistently yielded the highest error across all tests, yet it remains the most prevalent method in published literature.

The consequences of this reliance are not merely theoretical; they significantly inflate reported effect sizes. In a replication of high-profile studies, such as Williams et al. (2021), the researchers found that using the more accurate NSB estimator nearly halved the reported effect sizes. For example, the measured association between gendered nouns and adjectives in Polish—represented as Normalized Mutual Information (NMI)—dropped from 30.52% (MLE) to 17.07% (NSB). While the results remained statistically significant, the actual strength of the linguistic patterns was revealed to be much weaker than previously claimed, suggesting that a portion of measured "linguistic structure" is actually an estimation artifact.

Key Findings and Recommendations

The effectiveness of an estimator is highly dependent on sample size ($N$) and vocabulary size ($K$):

  • Low-Data Regimes: For very small samples ($N < 2000$), the Chao-Shen or Horvitz-Thompson estimators often perform best as they are designed to handle missing classes.
  • Large-Data Regimes: As the sample size grows, the NSB estimator becomes the superior choice, converging to the true entropy much faster than MLE or Miller-Madow.
  • Zipfian Sensitivity: Because language is Zipfian, estimators that do not account for the "long tail" fail spectacularly as the ratio of vocabulary size to sample size increases.

Conclusion

This research serves as a reality check for information-theoretic linguistics. To ensure robust and reproducible findings, researchers must move away from naive plug-in estimators. The study recommends a roadmap for future work: utilize the Chao-Shen estimator for limited datasets and the NSB estimator for larger corpora. While more computationally intensive, these methods provide the necessary accuracy to distinguish true linguistic patterns from the noise of finite sampling.

Summary updated

Abstract

Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropy must typically be estimated from observed data because researchers do not have access to the underlying probability distribution that gives rise to these data. While entropy estimation is a well-studied problem in other fields, there is not yet a comprehensive exploration of the efficacy of entropy estimators for use with linguistic data. In this work, we fill this void, studying the empirical effectiveness of different entropy estimators for linguistic distributions. In a replication of two recent information-theoretic linguistic studies, we find evidence that the reported effect size is over-estimated due to over-reliance on poor entropy estimators. Finally, we end our paper with concrete recommendations for entropy estimation depending on distribution type and data availability.
BibTeX
@inproceedings{arora2022estimating,
  author = {Aryaman Arora and Clara Meister and Ryan Cotterell},
  title = {{Estimating the Entropy of Linguistic Distributions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=GEAFNCg7XC5},
}

Loading related content...

Figure