Introduction
A central mystery in modern deep learning is why over-parameterized models—those with far more parameters than training examples—consistently achieve zero training error. While the Neural Tangent Kernel (NTK) framework provides a theoretical basis for this by analyzing networks in the infinite-width limit, it often fails to capture the behavior of finite-width networks used in practice. This paper reframes the problem by viewing two-layer neural networks as under-determined linear systems, providing new insights into the necessity of training hidden layers, the fundamental flaws of the ReLU activation function, and the benefits of kernel-based activations.
The Linear System Framework
The authors model a two-layer neural network as a linear system defined by the equation $y = A\alpha$. In this formulation, $A$ is a matrix determined by the input data, the activation function, and the fixed random weights of the hidden layer. The vector $\alpha$ represents the weights of the final output layer.
By treating the network this way, the researchers address three critical questions:
- Sufficiency: Is it necessary to train the entire network to reach zero loss, or is the last layer alone sufficient?
- Activation Limits: Are there inherent structural weaknesses in common activation functions like ReLU?
- Optimization Dynamics: How do the mathematical properties of these activations influence the speed of gradient descent?
The ReLU Blind Spot
One of the paper's most striking findings is the identification of a "blind spot" in standard ReLU networks. The authors prove that for a bias-free ReLU architecture, there exist specific datasets—some as small as eight well-separated points—that the network can never fit perfectly, regardless of how wide the hidden layer is. This occurs because the ReLU activation causes the underlying feature matrix $A$ to remain rank-deficient. This mathematical limitation suggests that the ubiquity of ReLU may be due to its simplicity rather than its numerical robustness.
Wendland Activations and Numerical Stability
To overcome the limitations of ReLU and the numerical instability of "smooth" activations like Swish, the authors introduce "Wendland activations." Borrowing from the theory of radial basis functions and Wendland kernels, these activations are mathematically guaranteed to produce positive definite matrices. This ensures that the linear system $y = A\alpha$ is always solvable.
Furthermore, the authors map the behavior of gradient descent onto Landweber iteration, a classical method for solving linear systems. They demonstrate that the speed of convergence is dictated by the "flatness" of the kernel matrix's eigenvalues. While Swish often leads to ill-conditioned matrices that cause gradient descent to stall, Wendland kernels maintain a flatter spectrum, allowing for significantly faster and more stable convergence.
Empirical Results
The researchers validated their framework across synthetic and real-world datasets, including Prostate-Cancer regression and Fashion-MNIST classification. Key results include:
- Solvability: On "stress test" datasets designed to break ReLU networks, Wendland activations maintained full rank and achieved zero loss where ReLU and Swish failed.
- Conditioning: Wendland activations produced condition numbers (a metric for numerical stability) that were orders of magnitude lower than those of traditional activations.
- Training Speed: On the Prostate-Cancer dataset, Wendland activations converged to zero error in tens of thousands of steps. In comparison, ReLU required over a million steps, and Swish failed to converge even after ten million steps.
- Width Bounds: The paper establishes that a network width of approximately $m \approx n \log n$ (where $n$ is the number of data points) is often sufficient to ensure the system is solvable by training only the last layer.
Conclusion and Future Directions
By viewing over-parameterized networks through the lens of linear algebra, this research demonstrates that training the final layer is often sufficient for convergence, provided the activation function is well-chosen. While this study focuses on two-layer networks and bias-free models, it highlights the spectral weaknesses of ReLU and the potential for kernel-inspired activations to improve training efficiency. Future work remains to extend these insights to deep, multi-layer architectures and to investigate how these findings impact generalization on unseen data.
Summary updated