A Truncation Model for Estimating Species Richness

Babagnidé François Koladjo, Mesrob I. Ohannessian, Élisabeth Gassiat

Introduction

Estimating species richness—the total number of distinct types within a population—is a fundamental challenge across diverse fields, from ecology and microbiology to linguistics and software bug detection. The central difficulty lies in accounting for "unseen" species: those that exist in the population but did not appear in the sample.

Historically, practitioners have addressed this using a "truncation" heuristic. They separate the data into "rare" species (those observed infrequently) and "abundant" species. The rare species are used to model the tail of the distribution and estimate the number of missing species, while the abundant species are simply counted. Despite its popularity, this practice has lacked a rigorous theoretical foundation, particularly regarding the choice of the truncation threshold ($\tau$) and the statistical efficiency of the resulting estimators.

The Semiparametric Truncation Model

This paper introduces a formal semiparametric framework that justifies and optimizes the truncation approach. The model divides the population into two components:

  1. The Rare Component: This is modeled parametrically, typically using a Poisson or Gamma-Poisson (Negative Binomial) distribution. This component captures the behavior of species with low discovery rates.
  2. The Abundant Component: This is modeled nonparametrically. By not assuming a specific functional form for abundant species, the model remains robust to the high variability often found in real-world data.

The authors assume that the support of the abundant component is "offset" from the rare range, meaning that truly abundant species are unlikely to appear as extremely rare observations (e.g., singletons).

Estimation Strategy

The authors derive a class of estimators using conditional maximum likelihood estimation (MLE). The process involves a stepwise maximization:

  • First, the parameters of the rare distribution are estimated.
  • The species richness ($N$) is then estimated as the integer that maximizes the binomial likelihood, given the other parameters.

A critical innovation in this work is the use of the Goldenshluger-Lepski (G-L) method to select the truncation threshold $\tau$. In previous methods, $\tau$ was often chosen arbitrarily (e.g., $\tau=10$). The G-L method provides a data-driven way to balance the bias of the parametric model against the variance of the estimator, allowing the model to adaptively determine where the "rare" regime ends.

Theoretical Significance

The proposed model offers two major theoretical breakthroughs:

  • Generalization of Chao’s Estimator: The authors demonstrate that their framework is a strict generalization of the widely used Chao lower bound. Specifically, when the rare component is a single-component Poisson and $\tau=2$, the estimator recovers the Chao result.
  • Asymptotic Efficiency: The paper proves that under regularity conditions, the estimators for the model parameters are asymptotically efficient. This provides the mathematical "gold standard" justification that was previously missing from truncation heuristics.

Empirical Results

The model's performance was evaluated through both synthetic simulations and a linguistic case study.

Simulation Studies

In scenarios involving high heterogeneity (where species abundances vary significantly), the proposed estimator ($\hat{N}_\tau$) consistently outperformed established methods. It achieved a lower relative Mean Square Error (rMSE) compared to five other common estimators, including penalized MLE and coverage-based methods. This suggests that the semiparametric approach is more robust to the "noise" of abundant species than purely parametric models.

Application: Molière’s Tartuffe

To demonstrate real-world utility, the authors applied the model to the vocabulary of the French playwright Molière. By treating unique words as "species," they estimated the total vocabulary size based on the text of the play Tartuffe. The model successfully predicted the vocabulary growth curve, showing that it could accurately infer the total "richness" of the author's lexicon from a limited sample.

Conclusion

This research provides a principled statistical basis for the common practice of data truncation in species richness estimation. By combining a parametric model for rare species with a nonparametric treatment of abundant ones, the authors have created a flexible, efficient, and theoretically sound estimator.

Future work remains to be done in developing robust confidence intervals using concentration inequalities and exploring more complex parametric forms for the rare component to further minimize bias in highly skewed populations.

Summary updated

Abstract

We propose a truncation model for the abundance distribution in species richness estimation. This model is inherently semiparametric and incorporates an unknown truncation threshold between rare and abundant observations. Using the conditional likelihood, we derive a class of estimators for the parameters in this model by stepwise maximization. The species richness estimator is given by the integer maximizing the binomial likelihood, given all other parameters in the model. Under regularity conditions, we show that our estimators of the model parameters are asymptotically efficient. We recover Chaos lower bound estimator of species richness when the parametric part of the model is single-component Poisson. Thus our class of estimators strictly generalized the latter. We illustrate the performance of the proposed method in a simulation study, and compare it favorably to other widely-used estimators. We also give an application to estimating the number of distinct vocabulary words in French playwright Molière's Tartuffe.
BibTeX
@article{koladjo2018truncation,
  author = {Babagnidé François Koladjo and Mesrob I. Ohannessian and Élisabeth Gassiat},
  title = {{A Truncation Model for Estimating Species Richness}},
  journal = {The International Journal of Biostatistics},
  year = {2018},
  doi = {10.1515/ijb-2017-0035},
}
Figure