Effect of Activation Functions on the Training of Overparametrized Neural Nets

Abhishek Panigrahi, Abhishek Shetty, Navin Goyal

Abstract

In the study of overparametrized neural networks, it is well-established that gradient-based methods can achieve near-zero training error with appropriate hyperparameters. While theoretical proofs for this phenomenon often assume ReLU activations, practical applications utilize a diverse array of functions such as tanh, swish, and ELU. This paper investigates how the choice of activation function impacts training speed by analyzing the minimum eigenvalue of the Gradient Gram matrix (G-matrix). We demonstrate a fundamental divide: non-smooth activations (e.g., ReLU, ELU) ensure fast convergence under minimal data assumptions, whereas smooth activations (e.g., tanh, swish) can lead to exponentially slow training in low-dimensional data settings. Furthermore, we show that increasing network depth can mitigate the training bottlenecks associated with smooth activations.

Introduction

Overparametrized neural networks—where the number of parameters significantly exceeds the number of training samples—have become the standard in modern machine learning. Despite the non-convex nature of the loss landscape, these networks are surprisingly easy to train using simple gradient descent. Theoretical explanations for this success typically rely on the properties of the Neural Tangent Kernel (NTK) or the Gradient Gram matrix. Specifically, the training speed is governed by the minimum eigenvalue ($\lambda_{min}$) of this matrix; a large $\lambda_{min}$ guarantees rapid convergence, while a value near zero indicates a plateau where training stalls.

While existing literature has provided rigorous bounds for ReLU networks, the theoretical understanding of other popular activation functions remains sparse. This paper addresses this gap by categorizing activation functions based on their smoothness and examining how this property interacts with data dimensionality to dictate training dynamics.

The Smoothness Divide

The core finding of this research is the distinction between "kinked" (non-smooth) and smooth activation functions.

1. Non-Smooth Activations (The "Kink" Advantage)

Activations like ReLU, SELU, and ELU possess points where their first or second derivatives are discontinuous. This non-smoothness is a critical feature rather than a flaw. The mathematical analysis reveals that the "jump" in the derivative prevents the G-matrix from becoming singular. For these functions, the authors prove that $\lambda_{min}$ remains large regardless of the intrinsic dimensionality of the input data, provided the data points are distinct. This ensures that the gradient remains informative throughout the training process.

2. Smooth Activations (The Smoothness Trap)

Smooth activations, such as tanh and swish, are infinitely differentiable. While they are often preferred for their mathematical elegance, they can be "too well-behaved." Using tools from approximation theory, including Hermite polynomials and Taylor expansions, the authors show that smooth functions can be closely approximated by low-degree polynomials.

If the input data lies in a low-dimensional subspace, the G-matrix for a smooth activation becomes nearly low-rank. In such scenarios, $\lambda_{min}$ can become "inverse superpolynomially small," leading to training speeds that are exponentially slower than those of ReLU networks. Essentially, in low dimensions, smooth activations struggle to distinguish between different data points, causing the optimization to stall.

The Role of Network Depth

Interestingly, the limitations of smooth activations are most pronounced in shallow (2-layer) networks. The paper demonstrates that depth acts as a "roughener." As signals pass through multiple layers of a smooth activation like tanh, the resulting composition of functions becomes increasingly complex and less globally smooth. This increased complexity helps boost the minimum eigenvalue of the G-matrix, allowing deep networks to overcome the dimensionality constraints that hinder shallow ones.

Empirical Validation

The theoretical framework was tested using both synthetic and real-world datasets:

  • Synthetic Data: By using points on a 1D circle ($S^1$) lifted into a higher-dimensional space ($S^9$), the researchers confirmed that smooth activations like tanh and swish converged significantly slower than ReLU and ELU when the intrinsic dimension was low.
  • CIFAR-10: On standard image classification tasks, ReLU consistently exhibited the highest eigenvalues and the fastest initial convergence, supporting the claim that non-smoothness facilitates more efficient optimization in the early stages of training.

Conclusion and Future Work

This research provides a rigorous mathematical basis for the empirical observation that activation functions significantly influence training efficiency. The results suggest that for tasks involving low-dimensional data or shallow architectures, non-smooth activations like ReLU or ELU are theoretically superior for ensuring fast convergence.

While this study offers a comprehensive look at training error within the NTK regime, several avenues for future work remain. The analysis focuses on the behavior of networks near their random initialization; exploring how these dynamics change as weights move further from initialization is a logical next step. Additionally, while this paper clarifies training speed, the relationship between activation smoothness and generalization error—the ability of the network to perform on unseen data—remains an open and vital question for the field.

Summary updated

Abstract

It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume that the activation function is ReLU or they crucially depend on the minimum eigenvalue of a certain Gram matrix depending on the data, random initialization and the activation function. In the later case, existing works only prove that this minimum eigenvalue is non-zero and do not provide quantitative bounds. On the empirical side, a contemporary line of investigations has proposed a number of alternative activation functions which tend to perform better than ReLU at least in some settings but no clear understanding has emerged. This state of affairs underscores the importance of theoretically understanding the impact of activation functions on training. In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks. A crucial property that governs the performance of an activation is whether or not it is smooth. For non-smooth activations such as ReLU, SELU and ELU, all eigenvalues of the associated Gram matrix are large under minimal assumptions on the data. For smooth activations such as tanh, swish and polynomials, the situation is more complex. If the subspace spanned by the data has small dimension then the minimum eigenvalue of the Gram matrix can be small leading to slow training. But if the dimension is large and the data satisfies another mild condition, then the eigenvalues are large. If we allow deep networks, then the small data dimension is not a limitation provided that the depth is sufficient. We discuss a number of extensions and applications of these results.
BibTeX
@inproceedings{panigrahi2020effect,
  author = {Abhishek Panigrahi and Abhishek Shetty and Navin Goyal},
  title = {{Effect of Activation Functions on the Training of Overparametrized Neural Nets}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=Qfgs8hfQRSS},
}
Figure