Recurrent architectures process sequential data by repeatedly applying a state transition function across time steps. Mathematically, this creates a deep computational graph where the hidden state at time $t$ depends heavily on the states before it.
Historically, this recurrent loop has forced a difficult design choice between two failure modes:
- Vanishing Gradients: Traditional activation functions like the hyperbolic tangent ($\tanh$) or logistic sigmoid saturate at their extremes. When a network processes long sequences, the derivatives of these functions approach zero, causing error signals to vanish as they backpropagate through time.
- Exploding Activations: Non-saturating activations like the Rectified Linear Unit (ReLU) keep gradients healthy because their derivative remains constant for positive values. However, repeatedly multiplying unbounded activations through a recurrent loop can cause the hidden state vectors to grow exponentially, leading to numerical instability and training failure.
To resolve this dilemma, a prominent research direction focuses on stabilizing non-saturating activations. By combining the gradient-preserving benefits of non-saturating functions with explicit regularization, architectural constraints, or bounded modifications, researchers have unlocked stable, deep, and fast recurrent sequence models.
Normalization and Explicit Regularization
The most direct way to prevent unbounded activations from exploding is to explicitly constrain their magnitude during training, either via auxiliary loss functions or normalization layers.
Instead of restricting the network's representational capacity, David Krueger et al. introduced a norm-stabilizer in Regularizing RNNs by Stabilizing Activations. This mechanism adds a penalty to the loss function based on the squared difference between the $L_2$ norms of successive hidden states:
$$\mathcal{L}_{\text{norm}} = \beta \frac{1}{T}\sum_{t=1}^T (\|h_t\|_2^2 - \|h_{t-1}\|_2^2)^2$$
This penalty prevents the activation magnitudes from exploding or decaying over time while leaving the network completely free to alter the direction of the state vector to encode information. While gated architectures naturally possess some degree of norm stability, the authors showed that explicitly enforcing it yields significant robustness.
Alternatively, stabilization can be built directly into the network's forward pass. Mirco Ravanelli et al. demonstrated in Improving Speech Recognition by Revising Gated Recurrent Units and Light Gated Recurrent Units for Speech Recognition that swapping $\tanh$ for a ReLU in a Gated Recurrent Unit (GRU) is highly effective if batch normalization is applied to the feed-forward transformations. The normalization rescales the pre-activations, providing the necessary regularization to counteract the unbounded nature of the ReLU.
Similarly, Arash Ardakani et al. utilized batch normalization after every vector-matrix multiplication in Learning Recurrent Binary/Ternary Weights to stabilize activation scales and prevent internal gates from saturating when using low-precision weights. For modern linear recurrent networks, Igor Dubinin et al. developed a joint $L_1$-gate normalization technique in Improved state mixing in higher-order and block diagonal linear recurrent networks, demonstrating that normalizing selective gates is essential for stabilizing training as block-diagonal recurrent architectures scale.
Architectural Constraints and Weight Regulation
Rather than relying on external normalization layers, another approach alters the structural properties of the recurrent cell itself to make non-saturating activations safe.
In standard RNNs, neurons within the same hidden layer are highly interconnected, making the eigenvalues of the recurrent weight matrix difficult to control. Shuai Li et al. bypassed this in Independently Recurrent Neural Network (IndRNN): Building a Longer and Deeper RNN by decoupling the neurons within each layer, effectively making the recurrent weight matrix diagonal. Because each neuron only receives its own past state, the authors could strictly regulate individual recurrent weights to prevent exponential gradient explosion. This structural constraint allowed them to safely integrate ReLU activations and train stable networks deeper than 20 layers.
Interestingly, the mathematical properties of ReLUs can sometimes provide intrinsic stability. Zeyuan Allen-Zhu et al. proved in On the Convergence Rate of Training Recurrent Neural Networks that at random initialization, ReLU activations naturally deactivate approximately half of the hidden neurons. This stochastic deactivation acts as a built-in buffer that stabilizes the hidden state norm across multiple time steps, preventing early gradient explosion or decay.
Other architectural innovations focus on stabilizing the gating mechanisms that control information flow. Sarath Chandar et al. showed in Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies that combining non-saturating units with explicit vector normalization ensures that the derivatives of the gating mechanisms never drop to zero. In a similar vein, Zhen Qin et al. introduced learnable lower bounds into sigmoid-based gating in Hierarchically Gated Recurrent Neural Network for Sequence Modeling, preventing the gates from saturating near zero or one and ensuring smoother optimization.
Alternative Non-Saturating and Bounded Functions
A third paradigm seeks a middle ground by designing custom activation functions that offer the gradient health of non-saturating functions but incorporate mathematical bounds to prevent numerical explosion.
Andrew Pulver et al. proposed a non-saturating logarithmic activation function in LSTM with working memory, defined as:
$$f(x) = \text{sign}(x) \cdot \ln(|x| + 1)$$
Unlike $\tanh$, which completely flattens out as values grow, this logarithmic curve continues to scale, maintaining healthy gradients even over long sequences. When paired with magnitude-based regularization, it provides a stable middle ground between saturating functions and the volatile linearity of ReLUs.
Conversely, piecewise linear bounded functions can be used to hard-cap state updates. Zhiyong Zhang et al. utilized a HardTanh activation for hidden state updates in NeuFlow-V2: Push High-Efficiency Optical Flow To the Limit, managing gradient flow effectively when states approach their limits. Biao Zhang et al. also emphasized in A Lightweight Recurrent Network for Sequence Modeling that keeping bounded activations like $\tanh$ in simplified recurrent structures is vital to prevent the state explosion seen in identity-based models.
Finally, shifting standard activations to different parts of the cell can alter stability. Habtamu Fanta et al. in SiTGRU: Single-Tunnelled Gated Recurrent Unit for Abnormality Detection and Habtamu Fanta et al. in "Forget" the Forget Gate: Estimating Anomalies in Videos Using Self-contained Long Short-Term Memory Networks found that replacing the traditional $\tanh$ candidate activation with a logistic sigmoid function mitigates activation degradation and improves convergence in deep networks. In the domain of State-Space Models (SSMs), Aleksandar Terzic et al. applied a sigmoid-based activation to the magnitude of diagonal transition components in Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models, ensuring bounded-input, bounded-output stability over long sequences.
Summary of Intuition
Why do these methods work? They decouple gradient health from numerical scale.
By ensuring that the derivative of the activation function does not decay to zero for large inputs, these techniques allow error signals to travel backward through hundreds or thousands of recurrent steps without vanishing. Simultaneously, by using norm penalties, batch normalization, independent neuron structures, or logarithmic curves, they prevent the forward-pass hidden states from growing exponentially. The result is a recurrent network that can learn long-term dependencies as effectively as a Transformer, but with the linear time complexity of a recurrent architecture.
Key Foundational Work
Regularizing RNNs by Stabilizing Activations (2016) — contributed 4 concept elements to this direction.
Regularizing RNNs by Stabilizing Activations David Krueger (2016) — contributed 3 concept elements to this direction.
On Fast Dropout and its Applicability to Recurrent Networks (2014) — contributed 1 concept elements to this direction.