Adaptive Huber Regression

Qiang Sun, Wen‐Xin Zhou, Jianqing Fan

Abstract

Big data can easily be contaminated by outliers or contain variables with heavy-tailed distributions, which makes many conventional methods inadequate. To address this challenge, we propose the adaptive Huber regression for robust estimation and inference. The key observation is that the robustification parameter should adapt to the sample size, dimension and moments for optimal tradeoff between bias and robustness. Our theoretical framework deals with heavy-tailed distributions with bounded $(1+\delta)$-th moment for any $\delta > 0$. We establish a sharp phase transition for robust estimation of regression parameters in both low and high dimensions: when $\delta \geq 1$, the estimator admits a sub-Gaussian-type deviation bound without sub-Gaussian assumptions on the data, while only a slower rate is available in the regime $0 < \delta < 1$.

Introduction

In the era of "big data," statistical models often rely on the "sub-Gaussian" assumption—the idea that data follows a well-behaved distribution without extreme outliers. However, real-world data in fields like finance, genetics, and fMRI studies frequently exhibit "heavy-tailed" distributions or contamination by outliers. When these anomalies occur, standard tools like Ordinary Least Squares (OLS) become highly unstable.

To address this, Sun, Zhou, and Fan introduce Adaptive Huber Regression, a framework that handles messy data without sacrificing the accuracy and efficiency of classical statistics. The core innovation is a version of Huber regression where the "robustness" parameter ($\tau$) automatically adapts to the sample size and data dimensionality, achieving an optimal tradeoff between bias and robustness.

Methodology

The Adaptive Huber estimator modifies the traditional Huber loss function, which is quadratic for small errors but linear for large ones. Unlike traditional approaches that use a fixed tuning parameter, this method scales $\tau$ dynamically.

Key technical components include:

  • Moment-Based Scaling: The framework accommodates distributions with only $(1+\delta)$-th moments for $\delta > 0$, a significantly weaker requirement than finite variance.
  • Regularization for High Dimensions: For settings where $d \gg n$, the authors integrate an $\ell_1$ penalty (Lasso-type) to perform simultaneous variable selection and robust estimation.
  • Computational Efficiency: The Local Adaptive Majorize-Minimization (LAMM) algorithm is employed. This iterative approach is computationally scalable for large datasets, avoiding the high costs of traditional interior-point optimization.
  • Robust Design: The theory extends beyond noisy responses to handle "heavy-tailed designs," where the predictor variables themselves may contain outliers.

Theoretical Insights: The Phase Transition

A central contribution of this work is the identification of a sharp "phase transition" in estimation accuracy based on the data's moment properties. This transition occurs at $\delta = 1$, the point where the second moment (variance) becomes finite.

Phase transition of the estimation error rate exponent $\zeta$ as a function of the moment parameter $\delta$.

As illustrated in the figure above, the estimation error rate (characterized by the exponent $\zeta$) behaves differently across two regimes:

  1. Regime $\delta \ge 1$: The estimator achieves a sub-Gaussian-type deviation bound ($\zeta = 1/2$), performing as efficiently as if the data were Gaussian, even without sub-Gaussian assumptions.
  2. Regime $0 < \delta < 1$: The estimation rate is slower ($\zeta = \delta/(1+\delta)$) but is proven to be the minimax optimal rate for that level of distribution "messiness."

By letting $\tau \to \infty$ as $n \to \infty$ at a specific rate, the estimator remains robust enough to ignore outliers in small samples while becoming asymptotically unbiased as the sample size grows.

Empirical Results

The authors validated their approach using both synthetic simulations and real-world genomic data.

Simulation Performance

In heavy-tailed scenarios using Student’s $t$ distributions, Adaptive Huber Regression (AHR) achieved a mean error of 0.806, significantly outperforming OLS (1.355). In extreme Log-normal cases, AHR (3.917) outperformed OLS (8.529) by more than double. Crucially, when data is normal, AHR loses almost no efficiency compared to OLS, making robustness essentially "free."

NCI-60 Cancer Cell Lines

On the NCI-60 dataset (59 observations, 17,924 predictors), the "Truncated" Adaptive Huber (TAHuber) model achieved the lowest Mean Absolute Error (5.76) compared to the standard Lasso (7.64). Furthermore, TAHuber selected a more concise and biologically relevant model (7 genes vs. 42 genes), identifying key genes like KRT19 and ANXA3 related to cancer cell migration that standard methods missed.

Conclusion

Adaptive Huber Regression provides a robust alternative to classical methods for heavy-tailed data. While the optimal $\tau$ depends on unknown moment parameters—requiring cross-validation or Lepski-type methods—the framework offers a powerful safety net. Future work may focus on sharpening high-dimensional bounds using sparsity-adaptive approaches like SLOPE and further optimizing computational overhead for massive data streams.

Summary updated

Abstract

Big data can easily be contaminated by outliers or contain variables with heavy-tailed distributions, which makes many conventional methods inadequate. To address this challenge, we propose the adaptive Huber regression for robust estimation and inference. The key observation is that the robustification parameter should adapt to the sample size, dimension and moments for optimal tradeoff between bias and robustness. Our theoretical framework deals with heavy-tailed distributions with bounded $(1+δ)$-th moment for any $δ&gt; 0$. We establish a sharp phase transition for robust estimation of regression parameters in both low and high dimensions: when $δ\geq 1$, the estimator admits a sub-Gaussian-type deviation bound without sub-Gaussian assumptions on the data, while only a slower rate is available in the regime $0
BibTeX
@article{sun2018adaptive,
  author = {Qiang Sun and Wen‐Xin Zhou and Jianqing Fan},
  title = {{Adaptive Huber Regression}},
  journal = {Journal of the American Statistical Association},
  year = {2018},
  doi = {10.1080/01621459.2018.1543124},
}
Figure