Abstract
In the study of overparametrized neural networks, it is well-established that gradient-based methods can achieve near-zero training error with appropriate hyperparameters. While theoretical proofs for this phenomenon often assume ReLU activations, practical applications utilize a diverse array of functions such as tanh, swish, and ELU. This paper investigates how the choice of activation function impacts training speed by analyzing the minimum eigenvalue of the Gradient Gram matrix (G-matrix). We demonstrate a fundamental divide: non-smooth activations (e.g., ReLU, ELU) ensure fast convergence under minimal data assumptions, whereas smooth activations (e.g., tanh, swish) can lead to exponentially slow training in low-dimensional data settings. Furthermore, we show that increasing network depth can mitigate the training bottlenecks associated with smooth activations.
Introduction
Overparametrized neural networks—where the number of parameters significantly exceeds the number of training samples—have become the standard in modern machine learning. Despite the non-convex nature of the loss landscape, these networks are surprisingly easy to train using simple gradient descent. Theoretical explanations for this success typically rely on the properties of the Neural Tangent Kernel (NTK) or the Gradient Gram matrix. Specifically, the training speed is governed by the minimum eigenvalue ($\lambda_{min}$) of this matrix; a large $\lambda_{min}$ guarantees rapid convergence, while a value near zero indicates a plateau where training stalls.
While existing literature has provided rigorous bounds for ReLU networks, the theoretical understanding of other popular activation functions remains sparse. This paper addresses this gap by categorizing activation functions based on their smoothness and examining how this property interacts with data dimensionality to dictate training dynamics.
The Smoothness Divide
The core finding of this research is the distinction between "kinked" (non-smooth) and smooth activation functions.
1. Non-Smooth Activations (The "Kink" Advantage)
Activations like ReLU, SELU, and ELU possess points where their first or second derivatives are discontinuous. This non-smoothness is a critical feature rather than a flaw. The mathematical analysis reveals that the "jump" in the derivative prevents the G-matrix from becoming singular. For these functions, the authors prove that $\lambda_{min}$ remains large regardless of the intrinsic dimensionality of the input data, provided the data points are distinct. This ensures that the gradient remains informative throughout the training process.
2. Smooth Activations (The Smoothness Trap)
Smooth activations, such as tanh and swish, are infinitely differentiable. While they are often preferred for their mathematical elegance, they can be "too well-behaved." Using tools from approximation theory, including Hermite polynomials and Taylor expansions, the authors show that smooth functions can be closely approximated by low-degree polynomials.
If the input data lies in a low-dimensional subspace, the G-matrix for a smooth activation becomes nearly low-rank. In such scenarios, $\lambda_{min}$ can become "inverse superpolynomially small," leading to training speeds that are exponentially slower than those of ReLU networks. Essentially, in low dimensions, smooth activations struggle to distinguish between different data points, causing the optimization to stall.
The Role of Network Depth
Interestingly, the limitations of smooth activations are most pronounced in shallow (2-layer) networks. The paper demonstrates that depth acts as a "roughener." As signals pass through multiple layers of a smooth activation like tanh, the resulting composition of functions becomes increasingly complex and less globally smooth. This increased complexity helps boost the minimum eigenvalue of the G-matrix, allowing deep networks to overcome the dimensionality constraints that hinder shallow ones.
Empirical Validation
The theoretical framework was tested using both synthetic and real-world datasets:
- Synthetic Data: By using points on a 1D circle ($S^1$) lifted into a higher-dimensional space ($S^9$), the researchers confirmed that smooth activations like tanh and swish converged significantly slower than ReLU and ELU when the intrinsic dimension was low.
- CIFAR-10: On standard image classification tasks, ReLU consistently exhibited the highest eigenvalues and the fastest initial convergence, supporting the claim that non-smoothness facilitates more efficient optimization in the early stages of training.
Conclusion and Future Work
This research provides a rigorous mathematical basis for the empirical observation that activation functions significantly influence training efficiency. The results suggest that for tasks involving low-dimensional data or shallow architectures, non-smooth activations like ReLU or ELU are theoretically superior for ensuring fast convergence.
While this study offers a comprehensive look at training error within the NTK regime, several avenues for future work remain. The analysis focuses on the behavior of networks near their random initialization; exploring how these dynamics change as weights move further from initialization is a logical next step. Additionally, while this paper clarifies training speed, the relationship between activation smoothness and generalization error—the ability of the network to perform on unseen data—remains an open and vital question for the field.
Summary updated