Abstract
How can neural networks such as ResNet efficiently learn CIFAR-10 with test accuracy more than 96%, while other methods, especially kernel methods, fall relatively behind? Can we provide theoretical justifications for this gap? Recently, there is an influential line of work relating neural networks to kernels in the over-parameterized regime, proving they can learn certain concept classes that are also learnable by kernels with similar test error. Yet, can neural networks provably learn some concept class BETTER than kernels? We answer this positively in the distribution-free setting. We prove neural networks can efficiently learn a notable class of functions, including those defined by three-layer residual networks with smooth activations, without any distributional assumption. At the same time, we prove there are simple functions in this class such that with the same number of training examples, the test error obtained by neural networks can be MUCH SMALLER than ANY kernel method, including neural tangent kernels (NTK). The main intuition is that multi-layer neural networks can implicitly perform hierarchical learning using different layers, which reduces the sample complexity compared to "one-shot" learning algorithms such as kernel methods. In a follow-up work, this theory of hierarchical learning is further strengthened to incorporate the "backward feature correction" process when training deep networks. In the end, we also prove a computation complexity advantage of ResNet with respect to other learning methods including linear regression over arbitrary feature mappings.
Introduction
In modern machine learning, deep neural networks like ResNet vastly outperform kernel methods on complex tasks. For instance, ResNet can achieve 96% accuracy on CIFAR-10, while the Neural Tangent Kernel (NTK) approach typically reaches only around 77%. Explaining this performance gap theoretically remains a significant challenge. While recent research has equated neural networks to kernels in the over-parameterized regime, this "NTK regime" fails to capture why neural networks generalize better. This paper addresses a fundamental question: Can we mathematically prove that neural networks efficiently learn a concept class with better generalization than kernel methods in a distribution-free setting?
Methods and Architecture
The authors utilize the framework of distribution-free PAC (Probably Approximately Correct) learning to analyze sample complexity. They define a specific class of target functions structured hierarchically: $\mathcal{H}(x) = \mathcal{F}(x) + \alpha\mathcal{G}(\mathcal{F}(x))$. In this structure, $\mathcal{F}$ represents a simpler "base" signal that contributes heavily to the output, while $\mathcal{G}$ is a more complex "composite" signal scaled by a small factor $\alpha$.
The study mathematically analyzes standard Stochastic Gradient Descent (SGD) training a 3-layer Residual Network (ResNet) with ReLU activations. To establish a baseline, they derive theoretical lower bounds for any kernel method (including NTK and random features) and any linear regression over arbitrary feature mappings.
Key Insights: Hierarchical Learning
The core finding is the mechanism of Forward Feature Learning. Unlike kernel methods, ResNets do not attempt to learn the entire function at once. Instead:
- Sequential Learning: Lower layers of the network automatically learn the simpler, lower-complexity base features ($\mathcal{F}$).
- Feature Forwarding: The network forwards these learned features via residual connections to higher layers.
- Composite Learning: Higher layers use these base features as a stepping stone to learn the higher-complexity composite features ($\mathcal{G}$).
In contrast, kernel methods suffer from a "one-shot" failure. They attempt to learn the composite function $\mathcal{G}(\mathcal{F})$ from scratch, which triggers the curse of dimensionality and requires an exponentially larger number of training samples.
Results and Empirical Validation
The theoretical analysis yields a provable separation in sample complexity. A 3-layer ResNet trained with SGD can learn the target concept class with $O(d/\alpha^8)$ samples (where $d$ is input dimension). Conversely, any kernel method requires at least $\Omega(d^k)$ samples, where $k$ is the polynomial degree of the complex function $\mathcal{G}$.
Empirical validation using synthetic datasets (combinations of parity functions) confirms these findings. While the ResNet successfully learned the target function, all tested kernel methods flatlined at high test error. Furthermore, the authors observed that SGD does not necessarily find the "simplest" solution in terms of Frobenius norm, suggesting that the architectural inductive bias of ResNet is more critical for generalization than simple norm minimization.
Limitations and Future Work
While this work proves a strict separation between ResNets and kernels, it contains an additive error term ($\alpha^4$) that prevents perfectly exact PAC-learning. This has been addressed in follow-up research through "backward feature correction." Additionally, while empirical evidence suggests a separation between ResNets and standard fully-connected networks, a formal theoretical proof for that specific architectural gap remains an open area for future research.
Summary updated