Abstract
We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.
Introduction: The Generalization Puzzle
Deep learning is arguably the most successful engineering feat of the 21st century, yet it remains a "black box" mathematically. Classical statistical learning theory, such as VC-dimension, suggests that as a model's complexity (number of parameters) increases, it should eventually overfit the training data and fail to generalize. However, deep learning defies this: practitioners find that larger, deeper models often perform better. This paper explores the emerging field of the mathematical analysis of deep learning, which seeks to explain why massive, overparametrized networks don't overfit and how they bypass the classical curse of dimensionality.
The Architecture of Deep Learning
At its core, a deep neural network is a composition of affine transformations and non-linear activation functions. Mathematically, a network with $L$ layers can be represented as a function $\Phi(x) = T_L \circ \rho \circ T_{L-1} \circ \dots \circ \rho \circ T_1(x)$, where $T_l(x) = W^{(l)}x + b^{(l)}$ are affine maps and $\rho$ is a non-linear activation function.
The choice of the activation function $\rho$ is critical for the network's expressivity. While classical models often used sigmoidal functions, modern deep learning heavily relies on the Rectified Linear Unit (ReLU) and its variants, which allow for efficient gradient propagation and the creation of complex, piecewise linear decision boundaries.
Key Mathematical Frameworks
To analyze these complex systems, researchers have developed several modern frameworks:
- Neural Tangent Kernel (NTK): This method analyzes DNNs in the "infinite width" limit. In this regime, training dynamics simplify into a linear regime (often called "lazy training"), allowing for convergence proofs using kernel methods.
- Approximation Theory: Functional analysis tools prove that deep networks can approximate complex functions (like "sawtooth" functions) exponentially more efficiently than shallow ones. Depth acts as a complexity multiplier, allowing a network to create a number of affine regions that grows exponentially with the number of layers.
- Implicit Regularization: This theory suggests that optimization algorithms like Stochastic Gradient Descent (SGD) inherently bias the model toward "simpler" solutions that generalize better, even in the absence of explicit penalties like L2 regularization.
- The Manifold Assumption: DNNs likely succeed because they don't learn functions in the full high-dimensional input space ($d$); instead, they learn functions on low-dimensional manifolds where the data actually resides, effectively beating the curse of dimensionality.
Key Insights and Results
One of the most significant findings is the Double Descent Curve. The classical "U-shaped" bias-variance trade-off is incomplete for deep learning. Beyond the "interpolation threshold" (where training error hits zero), the test error often drops again as the model becomes more overparametrized.
Furthermore, deep networks have shown unique capabilities in solving high-dimensional Partial Differential Equations (PDEs), such as the Schrödinger equation. They are currently the only known numerical tools capable of solving these problems without the computational cost exploding exponentially with the number of variables.
Limitations and Future Work
Despite rapid progress, theory still lags behind practice in several areas:
- Beyond Lazy Training: Most current proofs rely on the "lazy training" regime. A comprehensive theory for "active" training, where significant feature learning occurs, is still missing.
- Architecture Design: Many architectural choices, such as attention mechanisms in Transformers, remain heuristic and lack a rigorous mathematical foundation.
- Generalization Bounds: Existing generalization bounds are often too loose to be useful for practitioners. Developing "data-dependent" bounds that accurately predict test performance remains a primary goal for the field.
Summary updated