Standard neural networks are remarkably effective at interpolating smooth functions, but they struggle when forced to compute exact arithmetic or model discontinuous phenomena. When we introduce operations like division or flexible rational layers to solve these problems, we run into a fundamental mathematical barrier: singularities.
A singularity, or pole, occurs where a function's denominator approaches zero, causing its output to explode toward infinity. In a differentiable architecture trained via gradient descent, these points create highly volatile, spiky loss landscapes. Understanding why these singularities break optimization—and how to bypass them—is essential for building stable, mathematically expressive neural networks.
The Geometry of an Exploding Gradient
To understand why singularities crash training, consider the simplest reciprocal function: $f(x) = \frac{1}{x}$. As $x$ approaches zero from the positive side, the output $f(x)$ shoots up toward infinity.
However, the problem for neural networks is not just the forward pass; it is the backward pass. The derivative of this function is:
$$\frac{df}{dx} = -\frac{1}{x^2}$$
As $x$ approaches zero, the gradient explodes quadratically—far faster than the function value itself. If a network's weights or activations enter this region during optimization, a single backpropagation step will compute an astronomical gradient. This catapults the weights to extreme values, causing numerical overflow, catastrophic drift, or the dreaded NaN (Not a Number) errors.
Where Singularities Lurk
Singularities are not an isolated edge case; they emerge naturally across several prominent neural network paradigms:
1. Neural Division Modules
To generalize to numbers outside their training distribution, architectures require explicit arithmetic biases. While addition and multiplication are numerically well-behaved, division is inherently discontinuous at zero. As shown by Trask et al. and Mistry et al., integrating a differentiable division operator into neural arithmetic logic modules introduces severe optimization instability.
The authors of Madsen et al. demonstrated that this instability creates a spiky loss landscape that frequently disrupts the learning of other, simpler operations like multiplication. This risk extends beyond pure arithmetic modules; for instance, vector symbolic architectures that use element-wise division for unbinding operations face identical gradient explosions when vector components center near zero (Alam et al.).
2. Learnable Rational Activations
Instead of using fixed activation functions like ReLU, some architectures employ learnable rational functions (ratios of polynomials) to dynamically adapt the network's non-linearities. While highly expressive, Molina et al. and Trimmel et al. note that these layers are highly prone to training failures. Because the coefficients of the denominator polynomial are trainable, gradient descent can easily steer them into configurations where the denominator becomes zero for valid inputs, creating unexpected poles in the middle of the network.
3. Complex-Valued and Unitary Networks
Singularities also emerge from fundamental complex analysis. According to Liouville’s Theorem, any bounded complex function that is not constant must contain singularities where the function diverges (Minin et al.). Consequently, standard complex activations like the complex hyperbolic tangent exhibit periodic poles.
Even in real-valued variations like the modReLU activation function—frequently used in unitary recurrent neural networks to prevent vanishing gradients—a mathematical singularity occurs at the origin when the bias parameter is positive, causing gradients to explode to infinity and crash training (Maduranga et al.).
Why Naive Fixes Fail
The most intuitive way to prevent division by zero is to add a tiny constant $\epsilon$ to the denominator, modifying the operation to $\frac{1}{x + \epsilon}$. However, Puri et al. point out that this approach artificially caps the function's value, severely restricting the model's expressive capacity and preventing it from learning true asymptotic behaviors.
Alternatively, one can simply omit the division operator entirely to guarantee stability, as explored in certain neural arithmetic units (Madsen et al.). While this protects the loss landscape, it leaves the network incapable of representing fractions, ratios, or physical laws that rely on inverse relationships (Leite et al.).
Mechanisms for Stable Optimization
To successfully train networks containing division or rational layers, researchers have developed specialized architectural and optimization mechanisms designed to neutralize poles.
Curriculum-Based Regularization
Rather than enforcing a static boundary, Sahoo et al. introduced a regularized division function that actively suppresses both the outputs and the gradients when a denominator falls below a specific threshold.
Standard Division: [Denominator -> 0] ==> [Gradients Explode]
Regularized Division: [Denominator < T] ==> [Suppress Value & Gradient]
To allow the network to learn equations containing true poles, the authors utilize a curriculum that gradually lowers this threshold over the course of training. This allows the network to first find a stable global configuration in a smooth landscape before fine-tuning near the mathematical singularity.
Denominator Factorization
For rational activation functions, the risk of the denominator hitting zero can be mitigated by structural design. Trimmel et al. achieved stability by using a factorized denominator combined with partial fraction decomposition. By constraining the roots of the denominator polynomial mathematically, they prevent it from ever vanishing during gradient descent, ensuring smooth and reliable training.
Domain Splitting and Piecewise Substitutions
In complex-valued networks, where periodic singularities are mathematically guaranteed, architectures must actively avoid unstable regions of the complex plane. Minin et al. demonstrate that stability can be maintained by implementing piecewise substitutions near known poles, or by decoupling the operations entirely—processing the amplitude and phase of the complex input through separate, real-valued pathways.
Singularity-Aware Initialization
When a singularity sits at a fixed point—such as the origin in modReLU activations—the network can be protected purely through initialization strategies. As shown by Maduranga et al., initializing the network's weights and biases specifically to keep the hidden states away from zero prevents the model from encountering infinite gradients during the critical early phases of optimization.
Key Foundational Work
Learning Equations for Extrapolation and Control (2018) — contributed 3 concept elements to this direction.
Complex Valued Recurrent Neural Network: From Architecture to Training (2012) — contributed 3 concept elements to this direction.
Neural Arithmetic Logic Units (2018) — contributed 2 concept elements to this direction.