A Non-parametric Approach to Inference about the Tail of a Continuous or a Discrete Distribution

Jialin Zhang, Zhiyi ZhangarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a non-parametric, information-theoretic framework to classify and estimate the "thickness" of a distribution's tail. By leveraging the Good-Turing formula and a new concept called the "tail profile," the authors provide a visual and statistical toolkit (the TailClassifier R package) to distinguish between power-law, sub-exponential, and near-exponential decays in both discrete and continuous data.

What Problem Does This Paper Address?

Understanding the "tail" of a distribution—how frequently extreme events occur—is critical for risk management, insurance, and ecology. However, making inferences about tails is notoriously difficult for two reasons:

  1. Data Sparsity: By definition, tail events are rare, leaving researchers with very few data points to analyze.
  2. Model Mis-specification: Most standard tests require assuming a global parametric form (like a Normal distribution). If the tail behaves differently than the rest of the data, these models fail.

Existing tools, like the famous Hill estimator, are often restricted to power-law tails. There is a significant gap for a "model-agnostic" approach that can pinpoint the specific class of tail decay without making heavy prior assumptions.

Methods

The authors build their approach on Turing’s formula (or the Good-Turing estimator), originally developed by Alan Turing and I.J. Good to estimate the probability of "unseen" species in a sample.

Key components include:

  • Tail Profile ($\tau_v$): A set of information-theoretic quantities that act as a "fingerprint" for the tail. The authors provide an unbiased estimator ($Z_v$) for these indices.
  • Entropic Plots: This is the core diagnostic tool. By plotting the tail profile across different logarithmic scales, specific distribution families reveal themselves through linearity:
  • Plot 1 (log vs. log): Linear for Power-law tails (e.g., Pareto).
  • Plot 2 (log vs. log-log): Linear for Sub-exponential tails.
  • Plot 3 (log vs. log-log-log): Linear for Near-exponential tails.
  • The Binning Theorem: While the theory is rooted in discrete "alphabets," the authors prove that binning continuous data into equal-width intervals preserves the tail decay rate. This allows the method to be applied to continuous variables like stock returns or city populations.

Datasets

The paper validates the method using a mix of real-world and synthetic data:

  • Amazon (AMZN) Stock Returns: 30,000 minute-by-minute log returns used to classify the left (losses) and right (gains) tails.
  • 1950 US City Populations: A classic dataset originally used by Hill (1975) to demonstrate power-law behavior.
  • Simulated Distributions: Synthetic Power-law, Sub-exponential, and Near-exponential datasets used to test classification accuracy and parameter estimation consistency.

Key Insights and Ideas

  • Repurposing Species Estimation: The "aha moment" is realizing that the math used to predict missing species in biology is perfectly suited to describe the "missing" information in the sparse tails of probability distributions.
  • Linearity as a Classifier: Instead of complex p-values that are hard to interpret with small tail samples, the method relies on which "entropic plot" looks the most like a straight line. This provides a more intuitive, visual diagnostic for researchers.
  • Beyond Power Laws: Much of extreme value theory focuses on power laws (Zipf's law). This paper provides a rare, rigorous way to handle "thicker-than-exponential" but "thinner-than-power-law" distributions (sub-exponential and near-exponential).

Key Results

  • Classification Accuracy: In simulations, the method achieved near 100% correct classification rates for power-law and near-exponential tails, and ~79% for sub-exponential tails.
  • Revisiting Classics: When applied to the 1950 US City population data, the method suggested the tail might actually be sub-exponential rather than the power-law assumed by previous landmark studies.
  • Financial Risk: For Amazon stock returns, the method identified a near-exponential tail. This is a crucial distinction for risk managers, as it implies that "black swan" events are less likely than a power-law model would predict, but more likely than a standard Gaussian model suggests.

Limitations and Future Work

  • Index Selection: Choosing the "range" of the tail profile ($v_1$ to $v_2$) is currently a manual process. If the range is too small, there isn't enough info; if it's too large, the estimates become "noisy" and unreliable.
  • Computational Limits: Calculating the $Z_v$ estimator for very large values of $v$ can be computationally intensive, though the authors note that $v < 100$ is usually sufficient for most applications.
  • Automation: The authors suggest that future research could focus on automating the selection of the optimal tail profile range to make the tool more "plug-and-play" for non-statisticians.

Summary updated

Abstract

This article introduces a non-parametric information-theoretic approach to inference about the tail of a continuous or a discrete distribution. Leveraging a new concept named tail profile -- a set of information-theoretic quantities developed from results of domains of attraction on countable alphabets -- theoretical evidence supports the identification of specific discrete distributional tail types through a sequence of plots. The approach discerns tail types by bench-marking against exponential, and three thicker-than-exponential families: near-exponential, sub-exponential, and power-law (zipf, Pareto). For tails thicker-than-exponential, the approach also provides point and interval estimates for some of the underlying distribution parameters. While primarily designed to streamline the selection of discrete parametric models for detailed statistical analysis, a supporting theorem enables the method's extension use to continuous data, stating that binning continuous data with a common width preserves the tail decay rate under certain conditions. Simulations are presented to demonstrate the method's performance across various scenarios.
BibTeX
@misc{zhang2022non,
  author = {Jialin Zhang and Zhiyi Zhang},
  title = {{A Non-parametric Approach to Inference about the Tail of a Continuous or a Discrete Distribution}},
  year = {2022},
  eprint = {2204.12350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.12350},
}
Figure