New flexible versions of extended generalized Pareto model for count data

Touqeer Ahmad, Irshad Ahmad ArshadarXiv 2024 · DOI · arXiv

TL;DR

Researchers have developed new "extended" versions of the discrete generalized Pareto distribution (DEGPD) that can simultaneously model the common "bulk" of count data and its rare, extreme "heavy tails." These models eliminate the need for researchers to manually select a high threshold for extreme values, providing a more robust and flexible framework for analyzing everything from insurance complaints to hospital visits.

What Problem Does This Paper Address?

In the world of statistics, "count data" (integers like 0, 1, 2...) is everywhere. Traditionally, we use the Poisson or Negative Binomial distributions to model this data. While these work well for the "bulk" of the data—the average, everyday occurrences—they are notoriously bad at predicting "heavy tails," or rare, extreme events (the "Black Swans").

To handle these extremes, researchers often turn to the Discrete Generalized Pareto Distribution (DGPD). However, the DGPD has a major catch: it only works for data above a certain high threshold. Choosing this threshold is a massive headache—set it too low, and you bias your results; set it too high, and you don't have enough data points to be accurate. Furthermore, many real-world datasets are "zero-inflated," meaning they have an excessive number of zeros that standard tail-modeling tools struggle to reconcile with extreme values.

Methods

The authors propose a "best of both worlds" approach by creating Extended Discrete Generalized Pareto Distributions (DEGPD) and their zero-inflated counterparts (ZIDEGPD).

The core innovation lies in using an integral transformation. They take the standard Generalized Pareto Distribution (GPD) and "wrap" it in a mapping function, $G$. This mapping function is designed to satisfy specific mathematical conditions that allow it to control the "lower tail" (the bulk/zeros) without ruining the "upper tail" (the extremes) properties of the GPD.

They introduce three specific flavors of this mapping function:

  1. Model (i) / $M_1$: A Power Law mapping.
  2. Model (ii) / $M_2$: A Truncated Normal distribution mapping.
  3. Model (iii) / $M_3$: A Truncated Beta distribution mapping.

By discretizing these continuous functions, they created a toolset that can model the entire range of data—from zero to the most extreme outlier—in a single operation.

Datasets

The authors evaluated their models using both extensive simulations and three distinct real-world datasets:

  • AICCR (Automobile Insurance Company Complaint Rankings): Data from New York City used to test how well the models handle heavy tails in a general count setting.
  • Doctor Visits (DV): A dataset from the zic R package, characterized by a high proportion of zeros (zero-inflation) and a heavy tail.
  • Betting and Gaming Offenses: Data from New South Wales, Australia, specifically used to test the models' performance when forced to use a very low threshold for "extreme" events.

Key Insights and Ideas

  • Bypassing the Threshold: The biggest "aha" moment is that these models are "threshold-agnostic." Because they model the bulk and the tail together, researchers don't have to agonize over where the "extreme" part of the data begins.
  • Flexibility via $G$: The choice of the mapping function $G$ allows the model to adapt to different shapes of data. For instance, the Truncated Beta mapping ($M_3$) offers significant flexibility in the "bulk" of the distribution while keeping the tail estimates stable.
  • Stability at Low Thresholds: In the Australian crime dataset, the authors showed that while standard models (DGPD) failed or became biased when the threshold was set too low, the DEGPD models remained accurate. This is crucial for datasets where "extremes" aren't actually that far from the mean.

Key Results

  • Better Fit: Using the Bayesian Information Criterion (BIC), the proposed models (particularly $M_2$ and $M_3$) consistently outperformed standard models like the Zero-Inflated Negative Binomial (ZINB) and the basic DGPD.
  • Tail Accuracy: In the insurance dataset, all three DEGPD models converged on a similar "tail index" (approx. 0.73), showing they are highly reliable at capturing the severity of extreme complaints.
  • Zero-Inflation Mastery: In the doctor visits dataset, the ZIDEGPD models accurately estimated the proportion of zeros (around 38%) while simultaneously providing a robust fit for the high-frequency visitors in the tail.

Limitations and Future Work

While the models are powerful, the authors note that adding more parameters (like in the Beta-mapped model $M_3$) can sometimes lead to "identifiability problems"—essentially, the math gets so complex that it becomes hard for the computer to find the single "best" answer.

The authors suggest that the next frontier is Distributional Regression. This would involve linking the model's parameters (like the tail index or the zero-inflation probability) to other variables (covariates). For example, instead of just modeling "doctor visits," you could model how "doctor visits" change based on a patient's age, location, or insurance type, all while maintaining the ability to predict extreme outliers.

Summary updated

Abstract

Accurate modeling is essential in integer-valued real phenomena, including the distribution of entire data, zero-inflated (ZI) data, and discrete exceedances. The Poisson and Negative Binomial distributions, along with their ZI variants, are considered suitable for modeling the entire data distribution, but they fail to capture the heavy tail behavior effectively alongside the bulk of the distribution. In contrast, the discrete generalized Pareto distribution (DGPD) is preferred for high threshold exceedances, but it becomes less effective for low threshold exceedances. However, in some applications, the selection of a suitable high threshold is challenging, and the asymptotic conditions required for using DGPD are not always met. To address these limitations, extended versions of DGPD are proposed. These extensions are designed to model one of three scenarios: first, the entire distribution of the data, including both bulk and tail and bypassing the threshold selection step; second, the entire distribution along with ZI; and third, the tail of the distribution for low threshold exceedances. The proposed extensions offer improved estimates across all three scenarios compared to existing models, providing more accurate and reliable results in simulation studies and real data applications.
BibTeX
@misc{ahmad2024new,
  author = {Touqeer Ahmad and Irshad Ahmad Arshad},
  title = {{New flexible versions of extended generalized Pareto model for count data}},
  year = {2024},
  eprint = {2409.18719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.18719},
}
Figure