Rejoinder: “Nonparametric regression using deep neural networks with ReLU activation function”

Johannes Schmidt-HieberThe Annals of Statistics 2020 · DOI

In the evolving landscape of deep learning theory, a central debate persists: does the optimization process itself provide sufficient regularization, or do we need explicit structural constraints to ensure statistical reliability? In this rejoinder, Johannes Schmidt-Hieber defends the necessity of explicit regularization—specifically network sparsity—for achieving statistical consistency in nonparametric regression. While "implicit regularization" from Stochastic Gradient Descent (SGD) is often credited with the generalization success of overparametrized models, Schmidt-Hieber argues that it is frequently insufficient to filter out noise, potentially leading to estimators that fail to recover the true underlying function.

The Fundamental Tension: Interpolation vs. Consistency

Modern deep learning models often possess far more parameters than data points, allowing them to achieve zero training error—a phenomenon known as perfect interpolation. A popular hypothesis suggests that SGD biases these models toward "simple" solutions that generalize well despite overfitting the training set.

However, Schmidt-Hieber highlights a critical distinction between classification and regression. In classification tasks, where labels are often discrete and "clean," interpolation may be benign. In nonparametric regression, however, data is typically noisy. A model that perfectly interpolates noisy data points is essentially "fitting the noise." Schmidt-Hieber demonstrates that such models remain "wiggly" even as the sample size increases, preventing them from converging to the true, smooth regression function. This lack of convergence means the estimator is statistically inconsistent.

Theoretical Framework: Sparsity and Compositionality

To address the limitations of implicit regularization, the author utilizes mathematical proofs centered on network sparsity (limiting the number of non-zero weights). While practitioners may not always enforce strict sparsity, Schmidt-Hieber argues it serves as a vital theoretical benchmark:

  • Minimax Optimality: Sparsity allows for the derivation of "minimax" convergence rates—the optimal speed at which an estimator's error decreases as more data is provided.
  • Hierarchical Composition: Deep networks are uniquely suited to approximate functions built from simpler components (compositional functions). Sparsity provides the framework to prove that networks can adapt to this structure efficiently.
  • Overcoming the Curse of Dimensionality: In high-dimensional settings, traditional nonparametric methods often fail. Schmidt-Hieber shows that for additive models, deep ReLU networks can be structured such that complexity scales linearly rather than exponentially with the number of dimensions.

The Limits of Implicit Regularization

The rejoinder provides a two-step argument against relying solely on implicit regularization. By analyzing a smoothed, shallow ReLU network, the author proves that SGD converges to a variant of a natural cubic spline interpolant for one-dimensional inputs. Because this spline must pass through every noisy data point, it cannot be a consistent estimator of the underlying signal.

Furthermore, the paper suggests that the final state of a neural network is heavily dependent on its initialization and the specific trajectory of the optimizer. This reinforces the idea that the model architecture and the optimization algorithm are inextricably linked; one cannot guarantee consistency without considering both.

Empirical Grounding and Future Directions

While primarily a theoretical work, the paper aligns its findings with empirical phenomena like the "Lottery Ticket Hypothesis," which suggests that dense networks contain sparse sub-networks that can be trained to similar levels of accuracy. This supports the idea that sparsity is a relevant, if often latent, feature of high-performing models.

Looking forward, Schmidt-Hieber identifies several open challenges for the field:

  • Beyond Regression: Extending these consistency proofs to classification and other common deep learning tasks where the data distribution differs.
  • Algorithmic Bounds: Developing precise theoretical bounds for complex optimizers like Adam, which are currently less understood than simplified SGD.
  • Norm Control: Moving from sparsity (counting weights) to "norm control" (limiting weight magnitudes) as a more realistic proxy for how SGD regularizes modern networks.

Ultimately, the work serves as a cautionary note: while the "magic" of SGD is powerful, rigorous statistical guarantees in noisy environments likely require the deliberate structural constraints that explicit regularization provides.

Summary updated

BibTeX
@article{schmidthieber2020rejoinder,
  author = {Johannes Schmidt-Hieber},
  title = {{Rejoinder: “Nonparametric regression using deep neural networks with ReLU activation function”}},
  journal = {The Annals of Statistics},
  year = {2020},
  doi = {10.1214/19-aos1931},
}
Figure