Abstract
I would like to congratulate Johannes Schmidt–Hieber on a very interesting paper in which he considers regression functions belonging to the class of so-called compositional functions and analyzes the ability of estimators based on the multivariate nonparametric regression model of deep neural networks to achieve minimax rates of convergence. In my discussion, I will first regard such a type of result from the general viewpoint of the theoretical foundations of deep neural networks. This will be followed by a discussion from the viewpoint of expressivity, optimization and generalization. Finally, I will consider some specific aspects of the main result.
Introduction: The Theoretical Foundations of Deep Learning
Deep learning has revolutionized fields ranging from computer vision to healthcare, yet its success remains largely empirically driven. This "black box" nature poses significant challenges for sensitive applications where safety and reliability guarantees are paramount. In her discussion of Johannes Schmidt-Hieber’s landmark work, Gitta Kutyniok evaluates the progress made toward a rigorous mathematical foundation for deep neural networks.
Schmidt-Hieber’s original paper focuses on nonparametric regression using deep ReLU networks, specifically targeting "compositional functions"—functions structured as a hierarchy of simpler functions. While Kutyniok lauds the paper for its sophisticated linkage of expressivity and generalization, she argues that the theoretical framework remains incomplete. To truly understand deep learning, one must address what she terms the "Holy Trinity" of the field: Expressivity, Generalization, and Optimization.
The Three Pillars of Deep Learning Theory
Kutyniok frames the analysis of deep learning through three interconnected errors that constitute the total empirical risk:
- Expressivity Error: This concerns the approximation power of the network architecture. It asks whether a specific network structure (e.g., depth, width, activation function) can represent the target function class.
- Generalization Error: This involves the out-of-sample performance, or how well the model predicts unseen data based on a finite training set.
- Optimization Error: This is the error introduced by the training process itself, typically via algorithms like Stochastic Gradient Descent (SGD).
Schmidt-Hieber’s work primarily tackles the first two pillars. By utilizing sparsely connected ReLU networks and assuming a compositional structure for the regression functions, the paper demonstrates that deep networks can achieve minimax rates of convergence. However, Kutyniok points out that by ignoring the optimization process, the theory assumes we can always find the global minimizer, which is rarely the case in practice.
The "Naturalness" of Compositional Functions
A central theme in Schmidt-Hieber’s work is that compositional functions are "natural" for deep neural networks to learn. Kutyniok offers a nuanced critique of this perspective. She suggests that defining function classes based on what a specific tool (the neural network) is good at might be circular. Instead, she advocates for identifying realistic function classes derived from real-world data properties and then investigating whether neural networks are the optimal tool to approximate them.
Furthermore, while the compositional model helps explain how deep networks might circumvent the "curse of dimensionality," Kutyniok warns that the low-dimensional structure of these compositions might be "too easy." It may mask the true mechanism by which deep learning handles high-dimensional data without the exponential explosion of error typically seen in classical nonparametric statistics.
The Stochastic Reality of Optimization
One of the most significant contributions of Kutyniok’s discussion is the reminder that the trained network ($\hat{f}_n$) is not a fixed deterministic result. Because modern networks are trained using SGD, the resulting model is a stochastic term dependent on the randomness of mini-batch selection and initialization.
A complete theory must account for this stochasticity. Kutyniok argues that we need to understand the expected value of our models and how the choice of initial values influences the final convergence. Without integrating optimization error into the existing expressivity-generalization framework, the theory remains a "static" snapshot of a "dynamic" process.
Technical Critique: Theorem 1 and Practical Constraints
Kutyniok identifies a specific technical bottleneck in Schmidt-Hieber’s Theorem 1. The theorem requires the number of variables in the sub-functions ($t_i$) to be very small relative to the sample size ($n$) to maintain the desired convergence rates. In modern "Big Data" contexts, where $n$ can reach millions, this condition might be overly restrictive, potentially limiting the theorem's applicability to the architectures used in industry.
Despite these critiques, she acknowledges the strength of the generalization bounds provided. Schmidt-Hieber successfully established an upper bound for generalization error that explicitly depends on network depth ($L$) and the number of samples ($n$), providing a valuable roadmap for understanding the interplay between architecture and data volume.
Conclusion and Future Directions
Kutyniok’s discussion serves as a call to action for the theoretical community. To move beyond the current "empirically driven" era, future research must:
- Bridge the Optimization Gap: Integrate the training algorithm's convergence properties into the error bounds.
- Expand Loss Functions: Move beyond quadratic loss to include practical functions like hinge loss and incorporate standard regularization techniques like weight decay.
- Numerical Validation: Conduct extensive simulations to verify which theoretical estimates hold true under practical conditions.
By addressing these gaps, the field can move closer to a comprehensive theory that explains not just what deep networks can do, but how and why they do it so effectively.
Summary updated