A truncation model for estimating Species Richness

François Koladjo, Mesrob I. Ohannessian, Élisabeth Gassiat

Abstract

We propose a truncation model for abundance distribution in species richness estimation. This model is inherently semiparametric and incorporates an unknown truncation threshold between rare and abundant count observations. Using conditional likelihood, we derive a class of estimators for the model parameters via stepwise maximization. The species richness estimator is defined as the integer maximizing the binomial likelihood when all other parameters are known. Under regularity conditions, we demonstrate that these estimators are asymptotically efficient. Notably, we recover the Chao1 lower bound estimator when the model is a unicomponent Poisson model, establishing it as a specific element within our broader class of estimators. Simulation studies and applications to literary texts demonstrate the performance and robustness of the proposed method compared to existing benchmarks.

Introduction

Estimating species richness—the total number of species ($N$) in a population based on a sample of $D$ observed species—is a fundamental challenge in ecology, microbiology, and linguistics. Because sampling is rarely exhaustive, many species remain unobserved (the "zero-abundance" problem).

Statisticians often address this by partitioning data into "rare" species (those seen only a few times) and "abundant" species. The rationale is that rare species provide the most information about the number of missing species, while abundant species may follow complex distributions that bias standard parametric models. However, this partitioning is typically performed ad-hoc. This paper provides a formal semiparametric framework that justifies this truncation while maintaining theoretical guarantees of efficiency and consistency.

The Semiparametric Truncation Framework

The authors propose a mixture model where the probability of observing a species with abundance $x$ is defined as:

$$f(x) = q R_\theta(x) + (1-q) F(x)$$

In this model:

  • $R_\theta(x)$ represents the "rare" species component, modeled parametrically (e.g., Poisson or Negative Binomial).
  • $F(x)$ represents the "abundant" species component, modeled nonparametrically. Crucially, $F(x)$ is assumed to have no mass below a specific threshold $ au$.
  • $q$ is the proportion of rare species in the population.

Estimation Strategy

The researchers employ a conditional likelihood approach using stepwise maximization. They first estimate the parameters of the rare species distribution ($\theta$) and the mixture weight ($q$) using the observed data. Once these are established, they solve for the total population size $N$ by maximizing the binomial likelihood.

Two variations of truncation are discussed:

  1. Pure Truncation: Ignoring abundant species entirely.
  2. Hybrid Truncation: Using the total count of abundant species to weight the rare species' contribution. The authors prove that the hybrid approach is necessary for statistical efficiency unless the supports of rare and abundant species are completely disjoint.

To determine the optimal threshold $ au$, the authors adapt the Goldenshluger-Lepski (G-L) method. This heuristic balances the bias introduced by the parametric model against the variance of the estimator, allowing the model to adapt to the specific structure of the dataset.

Theoretical Insights

A significant contribution of this work is the unification of existing heuristics. The authors prove that the Chao1 estimator, a staple in ecological research, is a special case of their model. Specifically, when the rare species follow a simple Poisson distribution and the threshold is set to $ au=2$, their estimator recovers the Chao1 lower bound. This provides a new, rigorous theoretical foundation for one of the most widely used tools in the field.

Furthermore, by treating abundant species as a nonparametric "nuisance" component, the model gains robustness. It effectively treats abundant species as potential corruption; as long as they remain above the threshold $ au$, their specific distribution does not interfere with the estimation of the hidden (zero-abundance) species.

Empirical Validation

Synthetic Simulations

The model was tested against five major estimators (including Chao1 and various Maximum Likelihood Estimators) across populations ranging from 200 to 50,000 individuals. The proposed estimator, $\hat{N}_\tau$, consistently achieved the lowest relative Mean Square Error (rMSE), demonstrating superior accuracy and stability across different distribution shapes.

Literary Analysis: The Tartuffe Experiment

To test the model on real-world "observational richness," the authors analyzed the vocabulary of Molière’s play Tartuffe. In this context, "species" are unique word types and "abundance" is the frequency of those words. The model successfully predicted the total vocabulary size of the play even when provided with only 20-40% of the text, significantly outperforming standard lower-bound estimators.

Conclusion and Future Work

This research bridges the gap between heuristic data-splitting and formal statistical theory. By proving that their estimators reach the semiparametric efficiency bound, the authors provide a robust tool for researchers dealing with incomplete sampling.

While the G-L method for threshold selection performed well, the authors note that a formal proof of its optimality in this specific context remains an area for future study. Additionally, they suggest exploring more flexible "rare" models, such as those utilizing Chebyshev polynomials, to further refine the estimation of highly diverse ecosystems.

Summary updated

Abstract

We propose a truncation model for abundance distribution in the species richness estimation. This model is inherently semiparametric and incorporates an unknown truncation threshold between rare and abundant counts observations. Using the conditional likelihood, we derive a class of estimators for the parameters in the model by a stepwise maximisation. The species richness estimator is given by the integer maximising the binomial likelihood when all other parameters in the model are know. Under regularity conditions, we show that the estimators of the model parameters are asymptotically efficient. We recover the Chao$^{'}$s lower bound estimator of species richeness when the model is a unicomponent Poisson$^{'}$s model. So, it is an element of our class of estimators. In a simulation study, we show the performances of the proposed method and compare it to some others.
BibTeX
@article{koladjo2018truncation,
  author = {François Koladjo and Mesrob I. Ohannessian and Élisabeth Gassiat},
  title = {{A truncation model for estimating Species Richness}},
  journal = {The International Journal of Biostatistics},
  year = {2018},
  doi = {10.1515/ijb-2017-0035},
}
Figure