For years, the Rectified Linear Unit (ReLU) has been the default choice in deep learning due to its simplicity and sparse activations. However, its sharp "kink" at zero introduces non-differentiability and discontinuous gradients. While this might seem like a minor geometric detail, it fundamentally alters the loss landscape and limits the network's utility in domains requiring higher-order derivatives, continuous-depth modeling, or stable optimization trajectories.
By replacing piecewise linear functions with smooth, continuously differentiable alternatives—such as Softplus, GELU, Swish, or ELU—we can dramatically improve optimization geometry, physical consistency, and training stability across a wide variety of applications.
Smoothing the Optimization Landscape
Why does smoothing an activation function make networks easier to train? The answer lies in the geometry of the loss surface. Piecewise linear activations create a jagged, highly non-convex landscape filled with degenerate critical points. In contrast, analytic activation functions like Softplus or Sigmoid allow the application of differential topology and Morse theory. As shown by Quynh Nguyen et al. in The Loss Surface of Deep and Wide Neural Networks, when a network is sufficiently wide and utilizes smooth activations, degenerate critical points occur with zero probability, ensuring that almost all local minima are global minima.
Furthermore, this smoothness guarantees well-behaved first-order derivatives across the entire input space. Niladri S. Chatterji et al. provide a mathematical foundation in When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?, demonstrating that gradient descent with smooth approximations of ReLU (like Swish or Huberized ReLU) is theoretically guaranteed to achieve zero training loss and interpolate perfectly on separable data.
In deep architectures, Greg Yang demonstrates in Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization that non-collapsing smooth activations like SiLU or GELU prevent "dead zones" and feature collapse, allowing rich feature learning to persist across deep layers. Even in reinforcement learning, Zhifa Ke leverages twice-differentiable activations in An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks to establish tighter finite-time convergence bounds for deep Temporal Difference learning.
When Derivatives Matter: Physics and Control
The limitations of ReLU become glaringly obvious when a network's own derivatives are part of the optimization objective or control loop. Consider training an Implicit Neural Representation (INR) or computing volume rendering integrals. If the network uses ReLU, its first derivative is a step function, and its second derivative is zero everywhere except at the non-differentiable boundary.
To train networks defined by their derivatives, smooth non-linearities are essential. David B. Lindell et al. demonstrate in AutoInt: Automatic Integration for Fast Neural Volume Rendering that smooth activations maintain non-zero gradients and sufficient expressivity to learn closed-form antiderivatives for fast volume rendering. Similarly, Qingtian Zhu et al. show in Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives that periodic, infinitely differentiable activations are fundamentally better suited for Sobolev training (learning from gradient data) than piecewise linear alternatives.
In robotics and trajectory optimization, control signals are often derived directly from the model's gradients (the Jacobian matrix). Masood Dehghan et al. show in Analyzing Neural Jacobian Methods in Applications of Visual Servoing and Kinematic Control that continuously differentiable activations prevent the discontinuous control signals caused by ReLU, leading to more reliable and accurate robotic motion. This numerical stability is echoed by Armand Jordana et al. in Infinite-Horizon Value Function Approximation for Model Predictive Control and Elisa Alboni et al. in CACTO-SL: Using Sobolev learning to improve continuous actor-critic with trajectory optimization.
When modeling physical systems, such as the nuclear Equation of State, Kangmin Chen et al. emphasize in Neural Network Construction of the Equation of State from Relativistic ab initio Calculations that smooth activations prevent non-physical jumps in predicted values, maintaining thermodynamic stability.
The Continuous Limit: Neural ODEs and Invertible Networks
Smooth activations act as a crucial bridge between discrete architectures and continuous-depth models like Neural Ordinary Differential Equations (ODEs). In a Neural ODE, the hidden state evolves continuously over a vector field. If this field is parameterized by a non-smooth activation function, numerical ODE solvers encounter gradient inconsistencies and instability.
Tianxiang Gao et al. prove in Global Convergence in Neural ODEs: Impact of Activation Functions that smooth activations like Softplus ensure unique, stable solutions for both forward and backward passes, preventing solver divergence. From an architectural perspective, Pierre Marion et al. show in Implicit regularization of deep residual networks towards neural ODEs that smooth activations implicitly regularize deep Residual Networks, preventing layer-to-layer weight divergence and preserving the mathematical link to their continuous ODE limits.
For Invertible Neural Networks (INNs), Robin Chan et al. construct bijective activations in LU-Net: Invertible Neural Networks Based on Matrix Factorization by blending linear terms with Softplus, ensuring strict monotonicity and stable second derivatives for exact likelihood computation.
Stabilizing Adversarial Training and Generative Models
Minimax optimization problems, such as those found in Generative Adversarial Networks (GANs), are notoriously unstable. The choice of activation function directly dictates the spatial smoothness of the discriminator's gradients. Casey Chu et al. show in Smoothness and Stability in GANs that replacing ReLU with continuously differentiable alternatives like ELU or Sigmoid satisfies the mathematical conditions required for stable convergence. Without this smoothness, gradients can become chaotic, causing artifacts or "tearing" in the generated data distribution, a phenomenon analyzed by Alexander Korotin in the context of Wasserstein-2 Generative Networks.
In adversarial robustness, smooth activations prevent "gradient obfuscation"—where a model creates a jagged, non-linear loss surface to fool simple attacks but remains vulnerable to stronger ones. Chongli Qin et al. utilize smooth activations alongside local linearization in Adversarial Robustness through Local Linearization to ensure gradients remain predictable and well-behaved. For formal robustness evaluation, Huan Zhang et al. demonstrate in On Extensions of CLEVER: A Neural Network Robustness Evaluation Algorithm that twice-differentiable activations enable the use of higher-order Taylor expansions, allowing for nuanced landscape evaluations and formal guarantees.
Mathematical Consistency in Kernels and Sampling
Beyond standard optimization, smoothness is a prerequisite for many theoretical frameworks and advanced sampling techniques. For instance, analyzing networks via the Neural Tangent Kernel (NTK) requires mathematical consistency during gradient updates. Jonathan Hayase et al. use twice-differentiable activations like GELU in Few-shot Backdoor Attacks via Neural Tangent Kernels to ensure kernel-based approximations accurately predict network behavior during backdoor attacks, while Luke Eilers et al. employ smooth surrogate gradients in A generalized neural tangent kernel for surrogate gradient learning to prevent the NTK from becoming singular at training points.
Finally, in Bayesian Neural Networks, sampling from the posterior using Hamiltonian Monte Carlo (HMC) becomes highly inefficient under ReLU due to local integration errors at the non-differentiable boundaries. As proven by Vu C. Dinh et al. in Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient, the lack of gradient discontinuities in smooth networks allows for larger step sizes and higher acceptance rates, drastically improving sampling efficiency.