Traditional deep neural networks are explicit: data flows sequentially through a fixed stack of layers, where each layer computes its output directly from the previous layer's activations ($z_{t+1} = f_\theta(z_t, x)$).
Implicit models, such as Deep Equilibrium Models (DEQs), discard this rigid layer-by-layer progression. As outlined by Laurent El Ghaoui et al. in Implicit Deep Learning, an implicit layer defines its output $z^*$ as the solution to a fixed-point equilibrium equation:
$$z^* = f_\theta(z^*, x)$$
By finding a state where the representations stabilize, implicit models effectively simulate an infinitely deep network using the parameters of a single, weight-tied layer. Because the equilibrium state can be differentiated analytically using the Implicit Function Theorem, the backward pass does not need to store intermediate activations. This unlocks massive memory efficiencies, making implicit layers highly attractive for parameter-heavy tasks like object detection (Shuai Wang et al. in Deep Equilibrium Object Detection) and instance segmentation (Lufan Ma et al. in Implicit Feature Refinement for Instance Segmentation).
The "Speed Tax" of Equilibrium Convergence
Despite their memory elegance, implicit models suffer from a severe computational bottleneck: iterative solver latency.
Explicit Forward Pass:
[Input x] ---> [Layer 1] ---> [Layer 2] ---> [Layer 3] ---> [Output z]
Implicit Forward Pass:
+--- Loop until convergence (Iterative Solver) ---+
| |
v |
[Input x] ---> [Equilibrium Layer f_θ] --------------------------+---> [Output z*]
To compute the forward pass, an implicit network cannot simply execute a single mathematical transformation. Instead, it must run an iterative numerical root-finding algorithm (such as Anderson acceleration or Broyden's method) until the representation converges to $z^*$. If a model requires dozens of iterations to find a stable fixed point, its forward pass becomes orders of magnitude slower than a standard feed-forward network.
The bottleneck worsens during the backward pass. Computing exact gradients requires solving a massive linear system involving the inversion of the Jacobian matrix:
$$\left(I - \nabla_z f_\theta(z^*, x)\right)^{-1}$$
This dual burden—repeated function evaluations in the forward pass and expensive matrix operations in the backward pass—creates a significant latency penalty. A growing body of research focuses on breaking this bottleneck through learnable solvers, smarter initializations, and structural shortcuts.
Accelerating the Forward Pass
The standard approach to finding a fixed point treats the underlying function $f_\theta$ as a black box, relying on generic numerical solvers. Recent innovations replace these generic algorithms with context-aware mechanisms.
Learnable Hypersolvers
Shaojie Bai et al. introduce this concept in Neural Deep Equilibrium Solvers. Instead of using a standard root-finder, they train a tiny, lightweight neural network—a "hypersolver"—alongside the main model. This hypersolver learns the specific dynamical trajectories of the equilibrium function, predicting the path to the fixed point and cutting the number of required forward iterations in half.
Warm-Starting the Solver
Numerical solvers typically begin their search from a naive baseline, such as a zero vector. As noted by Shutong Ding et al. in Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation, starting far from the equilibrium state forces the solver to expend excessive function evaluations.
To bypass this, Weizhi Gao et al. propose a serialization strategy in Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing. By caching and reusing historical feature representations from nearby states, they "warm-start" the solver close to its final destination, yielding up to a 7× reduction in convergence latency.
Architectural Contraction Guarantees
If the weights of $f_\theta$ are poorly conditioned, the iterative solver can oscillate or diverge entirely. Hideaki Iiduka et al. tackle this in Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach by enforcing strict Lipschitz constraints directly on the network architecture. Ensuring that the layer behaves as a strict mathematical contraction guarantees rapid, stable convergence to a unique fixed point, drastically reducing the solver's workload.
Bypassing the Backward Pass Matrix Inversion
While accelerating the forward pass reduces inference latency, training remains throttled by the backward pass's Jacobian matrix inversion.
To alleviate this, Zhengyang Geng et al. introduce phantom gradients in On Training Implicit Models. They observe that computing the exact mathematical gradient is unnecessary during early and intermediate training phases.
Instead of solving the costly exact inverse system, phantom gradients substitute a lightweight approximation that skips the matrix inversion step entirely. This approximation decouples the backward pass from the exact equilibrium point, accelerating training wall-clock time by up to 1.7× while maintaining or even improving final task accuracy.
Collapsing and Eliminating the Equilibrium Loop
A final class of solutions re-engineers the equilibrium architecture to minimize or completely eliminate nested iterative loops.
Loop Merging
When implicit models are deployed in scenarios that require their own optimization loops—such as adversarial training or finding optimal latent codes—the latency compounds multiplicatively. In Joint inference and input optimization in equilibrium networks, the authors demonstrate that the internal fixed-point search of the DEQ can be mathematically merged with the external gradient-based input optimization loop. By optimizing both systems simultaneously within a single, augmented fixed-point system, they collapse two nested loops into one, achieving up to 9× speedups.
Explicit Equivalences
Can we enjoy the representational benefits of an implicit model without running an iterative solver at all? Two recent papers suggest that under certain conditions, the answer is yes:
- Koopman Operators: Takuya Konishi et al. introduce Stable Invariant Models via Koopman Spectra. They leverage Koopman operator theory to linearize and map the implicit dynamical system into an explicit, three-step feedforward network, bypassing root-finding entirely.
- Random Matrix Theory: In Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures, Zenan Ling et al. prove that in high-dimensional settings, the kernel properties of an infinitely deep implicit network match those of a carefully designed, shallow explicit network (consisting of only 1 or 2 layers).
These insights allow practitioners to match the performance of implicit architectures while executing a single, fast feed-forward pass.