In standard Stochastic Gradient Descent (SGD), we update our parameters by looking at the gradient where we currently stand. This "explicit" approach is the engine of modern machine learning, but it is fundamentally fragile. If the learning rate is slightly too high, or the loss landscape is too "stiff," the updates can overshoot, leading to numerical explosion and divergence.
A growing body of research suggests a more robust alternative: Implicit Stochastic Gradient Descent (ISGD). Instead of updating based on the current gradient, implicit methods define the next state based on the gradient at the next state. This subtle shift transforms optimization from a simple step into a self-correcting fixed-point problem that is "unconditionally stable."
The Fragility of Explicit Steps
To understand why implicit updates work, we must first see the failure mode of explicit ones. In a standard update, $\theta_{n+1} = \theta_n - \eta \nabla L(\theta_n)$, the step size $\eta$ must be carefully tuned. As Panos Toulis explains in Asymptotic and finite-sample properties of estimators based on stochastic gradients, if $\eta$ exceeds a threshold determined by the curvature (the Hessian) of the loss function, the system becomes unstable.
In complex models like Physics-Informed Neural Networks (PINNs) or Recurrent Neural Networks (RNNs), this "numerical stiffness" is pervasive. Songcan Chen, Ye Li, and Sheng-Jun Huang note in Implicit Stochastic Gradient Descent for Training Physics-Informed Neural Networks that standard optimizers like Adam often fail on differential equations because they cannot handle the high-frequency features that cause explicit steps to diverge.
The Implicit Fixed Point
The implicit update rule is defined as: $$\theta_{n+1} = \theta_n - \eta \nabla L(\theta_{n+1})$$
Notice that $\theta_{n+1}$ appears on both sides of the equation. This is a fixed-point iteration. While it is computationally more demanding to solve—often requiring a nested loop or an algebraic approximation—it offers a mathematical guarantee that explicit methods lack.
Panos Toulis and Edoardo M. Airoldi demonstrate in Stochastic gradient methods for principled estimation with massive data sets that this formulation is unconditionally stable. No matter how large you set the learning rate $\eta$, the update will not diverge. In fact, Songcan Chen showed that ISGD can use learning rates up to 500 times larger than standard methods without sacrificing stability.
Why It Works: The Contraction Effect
The intuition behind this stability is a "contraction" or "shrinkage" effect. In an explicit step, if the gradient is massive, the step is massive. In an implicit step, the algorithm "looks ahead." If a large step would land the parameters in a region with an opposing or even steeper gradient, the fixed-point equation naturally pulls the update back.
This is mathematically equivalent to a proximal operator. As Thomas Frerix, Michael Moeller, and Thomas Möllenhoff describe in Proximal Backpropagation, an implicit update is the solution to a local optimization problem: $$\theta_{n+1} = \arg\min_{\theta} \left( L(\theta) + \frac{1}{2\eta} \|\theta - \theta_n\|^2 \right)$$ The second term acts as a "proximity penalty." It ensures that the new parameters stay close to the old ones, effectively acting as a principled, data-adaptive trust region. This replaces the need for heuristic "gradient clipping" often used in deep learning.
Robustness in Reinforcement and Online Learning
The benefits of implicit updates extend beyond static supervised learning. In Reinforcement Learning, Aviv Tamar and Panos Toulis introduced Implicit Temporal Differences (TD) in Implicit Temporal Differences. They found that while standard TD is notoriously sensitive to step-sizes, the implicit version maintains the same computational efficiency ($O(k)$) while being significantly more robust to hyperparameter selection.
Similarly, in online learning, Chao-Bing Song and Tong Zhang proposed Fully Implicit Online Learning (FIOL) in Fully Implicit Online Learning. By performing exact updates for both the loss and the regularizer, they created algorithms that capture structural properties like sparsity much more effectively than linear approximations.
Scaling to Deep Networks
For a long time, implicit methods were considered too slow for deep learning because solving the fixed-point equation at every layer seemed impossible. However, Francois Fagan showed in Robust Implicit Backpropagation that layer-wise approximations can make these methods practical.
By approximating the implicit step, researchers like Nick Alonso and Emre Neftci have even linked these methods to biological "Inference Learning," suggesting that the brain might use similar self-correcting mechanisms to maintain stability during learning (A Theoretical Framework for Inference Learning).
The reader should take away that implicit updates are not just a different way to calculate a gradient; they are a different philosophy of movement. By accounting for the geometry of the destination before taking the step, they trade a small amount of per-iteration computation for a massive increase in global stability and hyperparameter robustness.