Lower bounds for the trade-off between bias and mean absolute deviation

Alexis Derumigny, Johannes Schmidt-Hieber

TL;DR

This paper provides a theoretical "reality check" for the idea that overparameterized models can bypass traditional statistical trade-offs. The authors prove that even when measuring stochastic error using Mean Absolute Deviation (MAD) instead of variance, a fundamental lower bound exists: you cannot reduce an estimator's bias without forcing a corresponding increase in its stochastic error.

What Problem Does This Paper Address?

In classical statistics, the "bias-variance trade-off" is a fundamental law: as you make a model more complex to reduce bias (systematic error), you inevitably increase variance (sensitivity to noise). However, recent breakthroughs in deep learning and "overparameterization" have challenged this view. Researchers have observed models that seem to achieve near-zero bias while maintaining low variance—a phenomenon often called "double descent."

This paper asks a deeper question: Is this trade-off specific to the way we measure error? Most classical theory uses squared error (variance). What if we use Mean Absolute Deviation (MAD), which is more robust and gives less weight to extreme outliers? Does the trade-off disappear, or is it a universal property of statistical estimation?

Methods

The authors approach this using the framework of nonparametric statistics, specifically focusing on the Gaussian white noise model. This is a standard theoretical "sandbox" used to study how well we can recover a smooth function from noisy data.

The core of their method involves:

  1. Abstract Inequalities: They derive new "change of expectation" inequalities. These are mathematical tools that relate how much the average value of an estimator changes when the underlying data distribution changes, compared to the estimator's MAD.
  2. Hellinger Distance: They use the Hellinger distance to quantify how "close" two probability distributions are.
  3. $\beta$-Hölder Smoothness: They assume the function being estimated belongs to a specific class of smooth functions (Hölder classes), which allows them to define what a "worst-case" scenario looks like for an estimator.

Unlike many papers that provide upper bounds (showing a specific method works well), this paper provides lower bounds, proving that no possible estimator can do better than a certain limit.

Datasets

As a theoretical statistics paper, this work does not use empirical datasets. Instead, it uses the Gaussian white noise model as a mathematical benchmark. This model serves as a proxy for high-dimensional regression problems, allowing the authors to prove universal truths that apply across many types of data.

Key Insights and Ideas

  • The Trade-off is Universal: The most important "aha moment" is that switching from variance to MAD doesn't save you. The authors show that the trade-off is a fundamental consequence of how information is extracted from noisy signals, not just a quirk of squared-error loss.
  • MAD vs. Variance: While MAD is more robust than variance, the authors establish that if you want an estimator with a "good" bias (converging at the optimal rate $n^{-\beta/(2\beta+1)}$), your MAD must also be at least that large. You cannot have a "negligible" bias without the MAD blowing up.
  • Information Geometry: The proof relies on the fact that for two functions to be distinguishable by an estimator, the "distance" between their probability measures (Hellinger distance) must be large enough. If the measures are too close, the estimator can't tell them apart, forcing a bias or a high MAD.
  • Implications for Confidence Bands: This has a direct impact on building confidence intervals. If you can't have an estimator with negligible bias, it means you can't easily build narrow, honest confidence bands for unknown functions without accounting for that bias.

Key Results

  • Theorem 4.1: This is the centerpiece. It states that for any estimator of a $\beta$-smooth function, if the worst-case bias is bounded by $C \cdot n^{-\beta/(2\beta+1)}$, then the worst-case MAD must be at least $c \cdot n^{-\beta/(2\beta+1)}$ for some constant $c$.
  • Rate Optimality: The results show that "rate-optimal" estimators—those that converge as fast as theoretically possible—must balance bias and MAD.
  • New Mathematical Tools: The authors provide a general-purpose inequality (Lemma 3.1) that relates the difference in expectations under two measures to the maximum MAD under those measures. This tool can be used by other researchers to study different statistical models.

Limitations and Future Work

  • Sharpness of Bounds: The authors note that their lower bound for MAD is "considerably weaker" (less mathematically tight) than the corresponding bounds for variance. There may be room to sharpen these constants in future work.
  • Gaussian Assumption: The main application is within the Gaussian white noise model. While this is a powerful proxy, extending these results to non-Gaussian noise or more complex loss functions (like those used in classification) remains an open area.
  • Beyond Pointwise Estimation: This paper looks at estimation at a single point ($x_0$). Future work could explore whether these trade-offs hold for "global" measures of error, like the integrated absolute error across the entire function.

Summary updated

Abstract

In nonparametric statistics, rate-optimal estimators typically balance bias and stochastic error. The recent work on overparametrization raises the question whether rate-optimal estimators exist that do not obey this trade-off. In this work we consider pointwise estimation in the Gaussian white noise model with regression function f in a class of β -Hölder smooth functions. Let ’worst-case’ refer to the supremum over all functions f in the Hölder class. It is shown that any estimator with worst-case bias ≲ n − β / ( 2 β + 1 ) ≕ ψ n must necessarily also have a worst-case mean absolute deviation that is lower bounded by ≳ ψ n . To derive the result, we establish abstract inequalities relating the change of expectation for two probability measures to the mean absolute deviation.
BibTeX
@inproceedings{derumigny2024lower,
  author = {Alexis Derumigny and Johannes Schmidt-Hieber},
  title = {{Lower bounds for the trade-off between bias and mean absolute deviation}},
  booktitle = {Statistics},
  year = {2024},
  doi = {10.1016/j.spl.2024.110182},
}

Loading related content...

Figure