Weak Signals and Heavy Tails: Machine-learning meets Extreme Value Theory

Stéphan Clémençon, Anne SabourinarXiv 2025 · DOI · arXiv

In the world of data science, we usually obsess over the "average" case. Most machine learning (ML) algorithms are designed to capture the statistical regularity of data near its center of mass. But what happens when the most important signals are buried in the "tails"—those rare, high-magnitude events like financial crashes, 100-year floods, or anomalous network intrusions?

Traditionally, Extreme Value Theory (EVT) handled these outliers using asymptotic, parametric models, while Statistical Learning Theory (SLT) provided the non-parametric, finite-sample guarantees for modern ML. In this survey paper, Stephan Clémençon and Anne Sabourin bridge the gap, offering a unified framework for learning from "weak signals" in "heavy tails."

TL;DR

This paper provides a theoretical bridge between Extreme Value Theory and Machine Learning, offering non-asymptotic guarantees for algorithms operating on rare events. It introduces a new high-dimensional regression tool called XLASSO and proves that common techniques like cross-validation actually work in the sparse, high-variance world of distribution tails.

What Problem Does This Paper Address?

Standard ML relies on concentration inequalities (like the VC inequality) that tell us how much an empirical average might deviate from the true mean. These work great when you have plenty of data. However, in the "tail" of a distribution, data is naturally scarce.

The authors address a fundamental tension:

  1. ML's Blind Spot: Classical algorithms ignore rare events because they contribute little to the "average" risk.
  2. EVT's Rigidity: Traditional EVT often requires strong parametric assumptions and only tells you what happens as your sample size goes to infinity.

The challenge is to create a non-parametric, non-asymptotic framework. How can we prove an algorithm will perform well on extreme data when we only have a handful of "extreme" examples ($k$) out of a large dataset ($n$)?

Methods

The authors leverage Multivariate Regular Variation, a mathematical property describing how the "shape" of a distribution stabilizes as you move further into the tails.

Key technical contributions include:

  • Tail Empirical Processes: Instead of looking at the whole dataset, they define measures specifically for observations exceeding a high threshold.
  • Angular Measures: They show that for many extreme problems, the "magnitude" of an event follows a predictable power law. The real "learning" happens in the angular component—the direction or pattern of the extreme features.
  • XLASSO (Extreme Lasso): An adaptation of the Lasso for high-dimensional regression where covariates are extreme. It uses an $\ell_1$ penalty on the angular components of the predictors.
  • Tail-Specific Concentration Inequalities: They derive new bounds where the error scales with $1/\sqrt{k}$ (where $k$ is the number of extreme points) rather than $1/\sqrt{n}$.

Datasets

The paper validates these theories using a mix of synthetic and real-world data:

  • Simulated Data: High-dimensional ($d=100$) additive noise models following multivariate symmetric logistic distributions.
  • Amazon & Yelp Reviews: Used to test LHTR (Learning Heavy Tailed Representations), showing that forcing word embeddings to follow EVT properties improves sentiment analysis on rare linguistic patterns.
  • Industry Portfolios Dataset: An open-access dataset (49 variables) used to demonstrate XLASSO's ability to predict sector-specific (e.g., Transportation) extremes based on other market factors.

Key Insights and Ideas

  • The $k$ vs. $n$ Trade-off: In standard ML, more data is always better. In EVT-ML, there is a new trade-off. If you pick too few extreme points ($k$), your variance is huge. If you pick too many, you include "bulk" data that biases your tail model.
  • Dimension Reduction via Angles: By focusing on the angular measure, the authors effectively reduce the dimensionality of the sample space by one, stripping away the predictable radial magnitude to focus on the complex dependence structure of the extremes.
  • LHTR & Data Augmentation: The authors suggest that we can "fix" standard embeddings (like BERT) by transforming them to satisfy EVT assumptions, and even generate synthetic "extreme" data points along tail curves to improve model robustness.

Key Results

  • XLASSO Superiority: In high-dimensional simulations ($d=100$), XLASSO significantly outperformed standard linear regression for tail prediction, especially when the number of extreme training points was small.
  • Cross-Validation Guarantees: The paper provides the first "sanity-check" guarantees for using CV in extreme regions, proving that CV-selected hyperparameters lead to models that generalize well even in low-probability zones.
  • Anomaly Ranking: The framework allows for more sophisticated anomaly detection that doesn't just flag "large" values, but identifies values that are "unusual" in their direction/pattern relative to other extremes.

Limitations and Future Work

  • Bounded Loss Constraints: The current theoretical guarantees for cross-validation require "bounded" loss functions. This excludes Extreme Quantile Regression, a major area of interest in finance and insurance that often uses unbounded losses.
  • Slow Rates for XLASSO: The current proofs for XLASSO provide "slow" convergence rates. The authors suggest that "fast rates" (common in standard Lasso theory) might be possible but would require new assumptions about the tail distribution of the angular components.
  • Threshold Selection: While the paper proves CV works, choosing the optimal threshold $k$ automatically remains a difficult practical challenge that warrants further algorithmic research.

Summary updated

Abstract

The masses of data now available have opened up the prospect of discovering weak signals using machine-learning algorithms, with a view to predictive or interpretation tasks. As this survey of recent results attempts to show, bringing multivariate extreme value theory and statistical learning theory together in a common, non-parametric and non-asymptotic framework makes it possible to design and analyze new methods for exploiting the scarce information located in distribution tails in these purposes. This article reviews recently proved theoretical tools for establishing guarantees for supervised or unsupervised algorithms learning from a fraction of extreme data. These are mainly exponential maximal deviation inequalities tailored to low-probability regions and concentration results for stochastic processes empirically describing the behavior of extreme observations, their dependence structure in particular. Under appropriate assumptions of regular variation, several illustrative applications are then examined: classification, regression, anomaly detection, model selection via cross-validation. For these, generalization results are established inspired by the classical bounds in statistical learning theory. In the same spirit, it is also shown how to adapt the popular high-dimensional lasso technique in the context of extreme values for the covariates with generalization guarantees.
BibTeX
@misc{clemencon2025weak,
  author = {Stéphan Clémençon and Anne Sabourin},
  title = {{Weak Signals and Heavy Tails: Machine-learning meets Extreme Value Theory}},
  year = {2025},
  eprint = {2504.06984},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.06984},
  howpublished = {arXiv},
}
Figure