Abstract
We propose a truncation model for abundance distribution in species richness estimation. This model is inherently semiparametric and incorporates an unknown truncation threshold between rare and abundant count observations. Using conditional likelihood, we derive a class of estimators for the model parameters via stepwise maximization. The species richness estimator is defined as the integer maximizing the binomial likelihood when all other parameters are known. Under regularity conditions, we demonstrate that these estimators are asymptotically efficient. Notably, we recover the Chao1 lower bound estimator when the model is a unicomponent Poisson model, establishing it as a specific element within our broader class of estimators. Simulation studies and applications to literary texts demonstrate the performance and robustness of the proposed method compared to existing benchmarks.
Introduction
Estimating species richness—the total number of species ($N$) in a population based on a sample of $D$ observed species—is a fundamental challenge in ecology, microbiology, and linguistics. Because sampling is rarely exhaustive, many species remain unobserved (the "zero-abundance" problem).
Statisticians often address this by partitioning data into "rare" species (those seen only a few times) and "abundant" species. The rationale is that rare species provide the most information about the number of missing species, while abundant species may follow complex distributions that bias standard parametric models. However, this partitioning is typically performed ad-hoc. This paper provides a formal semiparametric framework that justifies this truncation while maintaining theoretical guarantees of efficiency and consistency.
The Semiparametric Truncation Framework
The authors propose a mixture model where the probability of observing a species with abundance $x$ is defined as:
$$f(x) = q R_\theta(x) + (1-q) F(x)$$
In this model:
- $R_\theta(x)$ represents the "rare" species component, modeled parametrically (e.g., Poisson or Negative Binomial).
- $F(x)$ represents the "abundant" species component, modeled nonparametrically. Crucially, $F(x)$ is assumed to have no mass below a specific threshold $ au$.
- $q$ is the proportion of rare species in the population.
Estimation Strategy
The researchers employ a conditional likelihood approach using stepwise maximization. They first estimate the parameters of the rare species distribution ($\theta$) and the mixture weight ($q$) using the observed data. Once these are established, they solve for the total population size $N$ by maximizing the binomial likelihood.
Two variations of truncation are discussed:
- Pure Truncation: Ignoring abundant species entirely.
- Hybrid Truncation: Using the total count of abundant species to weight the rare species' contribution. The authors prove that the hybrid approach is necessary for statistical efficiency unless the supports of rare and abundant species are completely disjoint.
To determine the optimal threshold $ au$, the authors adapt the Goldenshluger-Lepski (G-L) method. This heuristic balances the bias introduced by the parametric model against the variance of the estimator, allowing the model to adapt to the specific structure of the dataset.
Theoretical Insights
A significant contribution of this work is the unification of existing heuristics. The authors prove that the Chao1 estimator, a staple in ecological research, is a special case of their model. Specifically, when the rare species follow a simple Poisson distribution and the threshold is set to $ au=2$, their estimator recovers the Chao1 lower bound. This provides a new, rigorous theoretical foundation for one of the most widely used tools in the field.
Furthermore, by treating abundant species as a nonparametric "nuisance" component, the model gains robustness. It effectively treats abundant species as potential corruption; as long as they remain above the threshold $ au$, their specific distribution does not interfere with the estimation of the hidden (zero-abundance) species.
Empirical Validation
Synthetic Simulations
The model was tested against five major estimators (including Chao1 and various Maximum Likelihood Estimators) across populations ranging from 200 to 50,000 individuals. The proposed estimator, $\hat{N}_\tau$, consistently achieved the lowest relative Mean Square Error (rMSE), demonstrating superior accuracy and stability across different distribution shapes.
Literary Analysis: The Tartuffe Experiment
To test the model on real-world "observational richness," the authors analyzed the vocabulary of Molière’s play Tartuffe. In this context, "species" are unique word types and "abundance" is the frequency of those words. The model successfully predicted the total vocabulary size of the play even when provided with only 20-40% of the text, significantly outperforming standard lower-bound estimators.
Conclusion and Future Work
This research bridges the gap between heuristic data-splitting and formal statistical theory. By proving that their estimators reach the semiparametric efficiency bound, the authors provide a robust tool for researchers dealing with incomplete sampling.
While the G-L method for threshold selection performed well, the authors note that a formal proof of its optimality in this specific context remains an area for future study. Additionally, they suggest exploring more flexible "rare" models, such as those utilizing Chebyshev polynomials, to further refine the estimation of highly diverse ecosystems.
Summary updated