Parameter-Free Robust Estimation for Heavy-Tailed Data

This direction is split into other directions: parameter-dependent robust estimation. Its last version stays readable and citable here.

Version 1 of 135 papers41 conceptsfrozen 2026-10-04

In classical statistics, we often assume that data follows a "well-behaved" distribution, like a Gaussian. In this world, the sample mean is the gold standard: it converges quickly and is easy to compute. However, real-world data—ranging from financial market fluctuations to signal processing—is rarely so polite. It frequently exhibits heavy tails, where extreme outliers are not just possible but expected.

When tails are heavy, the sample mean "breaks." A single massive outlier can pull the estimate far from the true center, and if the distribution has infinite variance, the sample mean may never converge at all. Recent research has focused on building a new generation of estimators that are robust to these outliers, with a specific emphasis on making them adaptive and parameter-free.

The Failure of the Sample Mean

The core problem with the sample mean is that it gives every data point equal weight. In a heavy-tailed setting, a single observation from the "tail" can be orders of magnitude larger than the mean. Justin Whitehouse, Ben Chugg, and Aaditya Ramdas note in Mean Estimation in Banach Spaces Under Infinite Variance and Martingale Dependence that this failure is even more pronounced in high-dimensional settings or when data points are dependent.

To fix this, researchers use three primary "robustification" strategies:

  1. Truncation: Throwing away or capping values that exceed a certain threshold.
  2. Median-of-Means (MoM): Splitting data into blocks, calculating the mean of each block, and taking the median of those means.
  3. M-Estimation (Catoni/Huber): Using a non-linear loss function (like the Huber loss) that "downweights" large values.

The Oracle Problem: Knowing the Tail

While these robust tools exist, they traditionally come with a catch: they require "oracle" knowledge. To set the optimal truncation threshold or the right block size for Median-of-Means, you usually need to know the distribution's tail index (how fast the probability of outliers decays) or a bound on its moments (e.g., "I know the 1.5-th moment is less than 10").

As Sujay Bhatt, Guanhua Fang, and Ping Li argue in Nearly Optimal Catoni's M-estimator for Infinite Variance, this dependency is a major barrier. In practice, we don't know these parameters beforehand. If we guess a moment bound that is too small, our estimator becomes biased; if we guess one that is too large, we lose robustness.

Achieving Parameter-Free Adaptivity

The "holy grail" of this field is an algorithm that can sense the "heaviness" of the data and adjust itself on the fly.

Adaptive Huber and Truncation

In Adaptive Huber Regression, Qiang Sun, Jianqing Fan, and Wen‐Xin Zhou demonstrated that the robustification parameter can be tuned adaptively to balance bias and robustness. Similarly, Yeshwanth Cherapanamjeri and Peter Bartlett established in Optimal Mean Estimation without a Variance that it is possible to match information-theoretic lower bounds even when the variance doesn't exist, provided the algorithm can adapt to the sample size and dimension.

Best-of-Both-Worlds Bandits

In the context of Multi-Armed Bandits (MAB), the challenge is even steeper because the algorithm must learn while it earns. Yu Chen and Yan Dai introduced uniINF in uniINF: Best-of-Both-Worlds Algorithm for Parameter-Free Heavy-Tailed MABs. This algorithm is "Best-of-Both-Worlds" (BoBW), meaning it performs optimally in both stable stochastic environments and volatile adversarial ones, all without knowing the tail parameters in advance.

Modern Frontiers: Change Points and Quantum Speedups

The research direction is now moving toward even more complex environments where the "rules" change over time.

Summary

The shift in this field is from robustness-by-assumption (where we assume we know the tail) to robustness-by-adaptation (where the algorithm learns the tail). By replacing the sample mean with adaptive M-estimators or data-driven truncation, researchers like Duo Cheng and Bo Ji are making it possible to deploy machine learning in high-stakes, volatile environments where outliers are the norm rather than the exception.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_free_robust_estimation_for_hea_e56a3033,
  title = {Parameter-Free Robust Estimation for Heavy-Tailed Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-free-robust-estimation-for-heavy-tailed-data/rdr_e56a3033d9a44d35aecf69ecd244f386},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_free_robust_estimation_for_hea_e56a3033_v1,
  title = {Parameter-Free Robust Estimation for Heavy-Tailed Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-free-robust-estimation-for-heavy-tailed-data/rdr_e56a3033d9a44d35aecf69ecd244f386/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}