Abstract
This paper investigates the theoretical properties of robust nonparametric estimation using deep neural networks (DNNs) for regression models characterized by heavy-tailed error distributions. We establish non-asymptotic error bounds for robust estimators using ReLU-activated DNNs under mild conditions on the error term, requiring only a finite $p$-th moment ($p > 1$). Furthermore, we demonstrate that these estimators can circumvent the curse of dimensionality when the predictor's distribution is supported on an approximate lower-dimensional set. Simulation studies confirm that robust methods significantly outperform standard least squares in the presence of heavy-tailed noise.
Introduction
In nonparametric regression, the objective is to recover an unknown function $f_0$ from noisy observations. While standard deep learning theory often assumes well-behaved, sub-Gaussian noise, real-world data frequently exhibits heavy-tailed distributions or outliers. In such scenarios, the conventional Least Squares (LS) approach, which minimizes Mean Squared Error (MSE), is highly sensitive to extreme values and often fails to provide reliable estimates. This work provides a rigorous foundation for using robust loss functions to overcome these challenges.
Robust Loss Functions and Methodology
To address the limitations of Least Squares, the authors analyze a variety of robust loss functions within the framework of feedforward neural networks using ReLU activation. These include:
- Least Absolute Deviation (LAD): Minimizes the $L_1$ norm of errors, providing inherent robustness to outliers.
- Huber Loss: A hybrid approach that behaves quadratically for small errors and linearly for larger ones, balancing efficiency and robustness.
- Quantile Loss: Used for estimating specific conditional quantiles of the response variable.
- Redescending Losses (Cauchy and Tukey’s Biweight): These losses are even more aggressive, with influence functions that decrease to zero for very large residuals, effectively ignoring extreme outliers.
The core of the research is a non-asymptotic error analysis. By relaxing the common assumption that the response variable $Y$ must be bounded, the authors prove that robust DNN estimators remain effective as long as the error distribution has a finite $p$-th moment for some $p > 1$.
Theoretical Insights: Bypassing the Curse of Dimensionality
A significant challenge in high-dimensional regression is the "curse of dimensionality," where the required sample size grows exponentially with the number of features. This paper provides a theoretical breakthrough by showing that DNNs can exploit low-dimensional structures in the data.
If the input data $X$ is concentrated near a lower-dimensional manifold of dimension $d_0$, the excess risk of the DNN estimator depends primarily on $d_0$ rather than the high-dimensional ambient space $d$. Specifically, for ReLU networks with appropriate width and size, the error bounds depend sub-linearly on $d$. This suggests that deep architectures are naturally suited for high-dimensional tasks where the underlying signal resides on a simpler manifold, even when that manifold is only approximately followed.
Empirical Performance
The theoretical findings are validated through extensive simulations using standard univariate benchmarks—Blocks, Bumps, Heavisine, and Doppler—and complex multivariate models. The models were tested against various noise profiles, including Standard Normal, Student’s $t$, Cauchy, and contaminated mixture models.
The results demonstrate a clear "breakdown" of the Least Squares method in the presence of heavy-tailed or mixture noise. In contrast, robust estimators like Huber and LAD maintain high accuracy. The Huber loss, in particular, emerged as a versatile choice, offering a strong balance between robustness and statistical efficiency across diverse noise conditions.
Conclusion
This work provides a rigorous foundation for the use of robust loss functions in deep nonparametric regression. By establishing that DNNs can handle heavy-tailed noise and high-dimensional data simultaneously, the authors bridge the gap between theoretical deep learning and practical, "messy" data analysis. While the current study focuses on outliers in the response variable, future work may extend these robust properties to handle contamination in the predictor variables as well.
Summary updated