Abstract
In nonparametric statistics, rate-optimal estimators typically balance bias and stochastic error. The recent work on overparametrization raises the question whether rate-optimal estimators exist that do not obey this trade-off. In this work we consider pointwise estimation in the Gaussian white noise model with regression function $f$ in a class of $\beta$-Hölder smooth functions. Let 'worst-case' refer to the supremum over all functions $f$ in the Hölder class. It is shown that any estimator with worst-case bias $\lesssim n^{-\beta/(2\beta+1)} =: \psi_n$ must necessarily also have a worst-case mean absolute deviation that is lower bounded by $\gtrsim \psi_n$. To derive the result, we establish abstract inequalities relating the change of expectation for two probability measures to the mean absolute deviation.
Introduction
In classical statistical theory, the bias-variance trade-off is considered an immutable law: as model complexity increases to minimize bias, the variance typically rises. However, the emergence of overparameterized deep learning models has challenged this intuition. Observations of "double descent"—where models with far more parameters than data points achieve near-zero training error and low generalization error—suggest that modern architectures might somehow bypass traditional trade-offs.
This paper provides a theoretical "reality check" for this notion. It investigates whether the trade-off is a byproduct of using squared-error loss (variance) or if it persists when using more robust measures of stochastic error, specifically Mean Absolute Deviation (MAD). By focusing on the Gaussian white noise model, the authors prove that the trade-off is a fundamental property of statistical estimation that cannot be avoided by simply changing the error metric.
The Gaussian White Noise Framework
To establish universal lower bounds, the study utilizes the Gaussian white noise model. This framework serves as a mathematical proxy for high-dimensional regression, allowing for the analysis of how well a smooth function $f$ can be recovered from noisy observations. The functions are assumed to belong to a $\beta$-Hölder smoothness class, which defines the regularity of the signal.
The goal is pointwise estimation at a specific point $x_0$. The authors define "worst-case" performance as the supremum of the error over all functions within the Hölder class. This approach ensures that the resulting bounds are not specific to a single "easy" function but represent the inherent difficulty of the estimation problem.
The Universality of the Bias-MAD Trade-off
The core contribution of this work is the proof that Mean Absolute Deviation (MAD) is subject to the same rate-optimal constraints as variance. In nonparametric statistics, the optimal convergence rate for a $\beta$-smooth function is $\psi_n = n^{-\beta/(2\beta+1)}$.
Theorem 4.1 establishes that if an estimator's worst-case bias is bounded by $C \cdot \psi_n$, then its worst-case MAD must be at least $c \cdot \psi_n$ for some constant $c$. In other words, one cannot achieve a "negligible" bias without the stochastic error (measured by MAD) remaining at the same order of magnitude. This confirms that rate-optimal estimators must balance these two components, regardless of whether the researcher prioritizes robustness against outliers.
Mathematical Foundations: Change of Expectation
The proof relies on a new set of abstract inequalities (specifically Lemma 3.1) that relate the change in an estimator's expectation across two different probability measures to the maximum MAD under those measures.
By employing Hellinger distance to quantify the proximity of two probability distributions, the authors show that if two functions are sufficiently close, an estimator cannot distinguish between them without incurring either significant bias or a high MAD. This information-geometric perspective demonstrates that the trade-off is rooted in the fundamental limits of information extraction from noisy signals.
Implications for Statistical Practice
These findings have significant implications for the construction of confidence bands. In nonparametric inference, building "honest" confidence intervals requires an accounting of the bias. If it were possible to create estimators with negligible bias and low MAD, one could theoretically construct narrower, more precise confidence intervals. This work proves that such a goal is mathematically impossible at the optimal rates, reinforcing the necessity of bias-aware inference.
Limitations and Future Directions
While the paper establishes a fundamental lower bound, the authors acknowledge several areas for further refinement:
- Sharpness of Bounds: The constants derived for the MAD lower bound are currently less tight than those known for variance. Future work may focus on sharpening these constants to provide more precise limits.
- Gaussian Assumptions: The current results are derived within the Gaussian white noise model. Extending these proofs to non-Gaussian noise distributions or different loss functions (such as those used in classification) remains an open challenge.
- Global vs. Pointwise Estimation: This study focuses on estimation at a single point. Investigating whether these trade-offs hold for global error measures, such as integrated absolute error across the entire domain, would further generalize the findings.
In conclusion, this research reaffirms that while overparameterization may change how we approach model fitting, it does not exempt us from the fundamental mathematical trade-offs between systematic and stochastic error.
Summary updated