Threshold Selection for Extreme Value Estimation

115 concepts83 papers1 proposal

When modeling rare, catastrophic events—such as financial market crashes, once-in-a-century floods, or severe pandemic outbreaks—standard statistical tools often fail. Because these events are by definition scarce, we cannot rely on traditional averages. Instead, we turn to Extreme Value Theory (EVT), which provides a mathematical framework to model the "tails" of probability distributions.

The most widely used approach in EVT is the Peaks-over-Threshold (POT) method, which isolates and analyzes only the data points that exceed a certain high cutoff. But this introduces a deceptively difficult question: Where exactly does the tail begin?

Choosing this boundary, or threshold, exposes a fundamental bias-variance tradeoff that lies at the heart of modern extreme value statistics.


The Anatomy of the Tradeoff

To understand why selecting a threshold is so difficult, imagine trying to estimate the shape of a distribution's tail using a finite dataset.

Low Threshold -----------------------> High Threshold
(Large sample size, high bias)         (Small sample size, high variance)
Includes non-tail data                 Pure tail data, but too few points

If you set the threshold too low, you include a large number of observations. While a larger sample size reduces the variance of your estimator, it introduces data from the central bulk of the distribution. Asymptotic models like the Generalized Pareto Distribution (GPD) only accurately describe data at the absolute limit of the tail. Incorporating non-extreme data violates these asymptotic assumptions, creating a systematic model approximation error, or bias. As noted by Hu et al. in Modeling multivariate extreme value distributions via Markov trees, lower thresholds pull the model away from true tail behavior by forcing it to fit the bulk of the data.

Conversely, if you set the threshold too high, you ensure that the remaining data points purely reflect true tail behavior. However, you throw away the vast majority of your dataset. With only a handful of extreme observations left, the statistical variance skyrockets. In On Binary Classification in Extreme Regions, Jalalzai et al. emphasize that this data sparsity makes high-threshold models incredibly sensitive to random noise, leading to highly unstable predictions.


Quantifying the Error via Higher-Order Expansions

To move beyond qualitative intuition, researchers use advanced mathematical mappings to track exactly how bias accumulates as the threshold changes.

In Assessing confidence intervals for the tail index by Edgeworth expansions for the Hill estimator, the authors (including Johan Segers) utilize higher-order Edgeworth expansions to dissect the behavior of the Hill estimator—the standard tool for measuring heavy-tailed risks. Their mathematical map reveals a striking danger: when you include too many observations, the resulting confidence intervals appear deceptively narrow and precise. In reality, however, the underlying bias shifts the entire interval away from the true value, rendering standard statistical conclusions completely misleading.

This phenomenon is further detailed by Troop et al. in Bias-corrected peaks-over-threshold estimation of the CVaR. They demonstrate that the approximation error from a slightly misaligned threshold propagates heavily when estimating risk metrics like Conditional Value-at-Risk (CVaR), proving that the boundary between the body and the tail cannot be chosen arbitrarily.


Data-Driven Solutions: Finding the Sweet Spot

Historically, practitioners relied on manual, visual heuristics—such as looking for stable plateaus in "Hill plots"—to pick a threshold. However, these methods lack reproducibility. Modern research focuses on automated, data-driven strategies to optimize this tradeoff.

The Bootstrap Approach

One elegant solution is to let the data find its own optimal balance by minimizing the Mean Squared Error (MSE). In Regenerative block-bootstrap confidence intervals for tail and extremal indexes, Bertail et al. show how resampling techniques can be used to estimate the MSE across a spectrum of potential thresholds. By simulating alternative realizations of the data, the bootstrap algorithm identifies the exact cutoff point where the sum of squared bias and variance is minimized.

The Parameter Stability Approach

Another core intuition is that if a threshold has truly entered the tail region, the estimated tail parameters should remain relatively constant even if the threshold is pushed even higher. In Peaks Over Thresholds Modeling With Multivariate Generalized Pareto Distributions, Kiriliouk et al. operationalize this by tracking multivariate asymptotic dependence measures across high quantiles. The point at which these measures stabilize provides a robust, data-driven criterion for isolating extreme data.


Mitigating the Tradeoff via Explicit Bias Correction

Rather than merely searching for a compromise threshold, an alternative strategy is to mathematically model and subtract the bias itself.

By incorporating a second-order asymptotic parameter that captures how fast the distribution converges to the tail limit, researchers can build bias-reduced estimators. For instance, Kaibuchi et al. introduce a framework called GARCH-UGH in GARCH-UGH: A bias-reduced approach for dynamic extreme Value-at-Risk estimation in financial time series. This method explicitly corrects for the systematic distortion introduced by lower thresholds, effectively allowing risk managers to utilize more data points to lower variance without suffering from the typical penalty of high bias.


Alternative Paradigms: The Block Maxima Problem

Does shifting away from threshold-based methods eliminate this tension? Not entirely. The other primary branch of EVT is the Block Maxima method, which divides a time series into equal blocks (e.g., years) and fits a Generalized Extreme Value (GEV) distribution to the maximum values of those blocks.

As analyzed by Bücher et al. in Maximum likelihood estimation for the Fréchet distribution based on block maxima extracted from a time series, this paradigm simply translates the threshold problem into a block size problem:

  • Small blocks yield more blocks (more data points, lower variance) but fail to reach the true asymptotic limit, introducing severe bias.
  • Large blocks satisfy the asymptotic theory perfectly (low bias) but drastically reduce the number of available maxima, driving up variance.

Ultimately, whether partitioning data via thresholds or blocks, the core challenge remains identical: extreme value analysis requires navigating a delicate boundary where mathematical limits meet the constraints of finite data.


Why It Matters for Modern Machine Learning

Resolving the threshold selection problem is no longer just an exercise for theoretical statisticians; it has become vital for high-dimensional machine learning and causal inference.

When dealing with functional data streams, such as energy grid spikes, Huet et al. demonstrate in Regular Variation in Hilbert Spaces and Principal Component Analysis for Functional Extremes that dimension-reduction techniques like PCA are highly sensitive to the chosen tail threshold. Similarly, in causal inference, estimating how a policy or treatment impacts rare, worst-case outcomes requires robust thresholding. As Tan et al. show in Estimation of Treatment Effects in Extreme and Unobserved Data, developing automated, theoretically grounded threshold selection rules is the key to unlocking stable, reliable causal estimates in regions where data is naturally scarce.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 77 papers
  2. Version 1 · 2026-10-04 · 100 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_extreme_value_threshold_selection_b63f534f,
  title = {extreme value threshold selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extreme-value-threshold-selection/rdr_b63f534ffecb44c1b7fcf77d034e233a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_extreme_value_threshold_selection_b63f534f_v2,
  title = {extreme value threshold selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extreme-value-threshold-selection/rdr_b63f534ffecb44c1b7fcf77d034e233a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake