In the standard paradigm of deep learning, "deeper is better." To increase the representational power of a network, researchers typically stack more layers—moving from ResNet-50 to ResNet-101 and beyond. However, this depth comes at a significant cost: memory. During training, standard backpropagation requires storing the intermediate activations of every single layer to compute gradients. This linear relationship between depth and memory consumption creates a hard ceiling for scaling models, especially in high-resolution computer vision or long-sequence modeling.
Deep Equilibrium Models (DEQs) and the broader class of implicit networks offer a way out of this bottleneck. Instead of defining a network as a sequence of discrete transformations, these models define their output as the fixed point of a single, recursive transformation.
The Equilibrium Perspective
In a traditional feedforward network, the output is computed as $h_{i+1} = f(h_i, x)$. In contrast, Shaojie Bai, J. Zico Kolter, and Vladlen Koltun proposed the Deep Equilibrium Model, which searches for a state $z^*$ such that:
$$z^* = f(z^*, x)$$
Here, $f$ is a weight-tied layer (or a small block of layers) that is applied repeatedly until the hidden state $z$ stabilizes. This is conceptually equivalent to an infinite-depth network where every layer is identical. Because the model iterates until convergence, it naturally finds the "steady state" of the representation for a given input $x$.
This framing has been extended to various domains. For instance, Fangda Gu, Laurent El Ghaoui, and colleagues introduced Implicit Graph Neural Networks, which capture infinitely long-range dependencies in graphs by solving for an equilibrium node representation, avoiding the "oversmoothing" issues seen in traditional deep GNNs.
Constant Memory via Implicit Differentiation
The most transformative aspect of this research is how these models are trained. Normally, to train an iterative process, one would "unroll" the iterations and backpropagate through each step. This is memory-intensive and numerically unstable for many iterations.
DEQs bypass this by using the Implicit Function Theorem (IFT). The IFT allows us to compute the gradient of the loss with respect to the model parameters directly at the equilibrium point $z^*$, without knowing the path the solver took to get there. As Minsik Lee and Jin Young Choi 0002 describe in their work on Differentiable Fixed-Point Iteration Layers, this reduces the memory cost of training to a constant—$O(1)$—relative to the "depth" of the iterations.
Whether the model takes 10 iterations or 1,000 to converge, the memory required for backpropagation remains the same because intermediate activations are never stored. This decoupling of depth from memory has enabled the training of Multiscale Deep Equilibrium Models that achieve state-of-the-art performance on vision tasks like ImageNet while using up to 88% less memory than comparable feedforward models.
Solving Inverse Problems and Optimization
The fixed-point formulation is particularly natural for problems that are already iterative in nature, such as physics simulations or image reconstruction.
In imaging, Davis Gilton and Rebecca Willett demonstrated that Deep Equilibrium Architectures for Inverse Problems can replace "deep unrolling" (which uses a fixed, small number of layers). By seeking a true equilibrium, these models allow for a trade-off at test time: one can run the solver longer for higher precision without retraining the model. Similarly, C. J. Daniele and Silvia Villa applied this to Poisson imaging by integrating Mirror Descent into the DEQ framework, leveraging non-Euclidean geometry to handle photon-limited data.
The reach of this idea extends to game theory and distributed systems. Howard Heaton and Wotao Yin developed Nash Fixed Point Networks, which use operator splitting to predict equilibria in convex games. By treating the Nash equilibrium as a fixed point, they can train models to predict agent behavior in large-scale systems like traffic routing.
Stability and Efficiency Improvements
While the IFT provides a theoretical path to gradients, practical implementation requires efficient root-finding and Jacobian handling.
- Jacobian-Free Methods: To avoid the high cost of inverting large Jacobian matrices, researchers like Naoki Murata have utilized Jacobian-free backpropagation, which approximates the necessary terms using iterative solvers (like the Broyden method) or Neumann series.
- Stochastic Optimization: Jiaming Liu and Weijie Gan introduced Online Deep Equilibrium Learning, showing that integrating minibatch processing directly into the DEQ solver can lead to 3x speedups in training for image restoration.
- Refinement and Stability: In the context of object representation, Michael Chang and Sergey Levine showed that treating object slots as fixed points makes iterative refinement significantly more stable, removing the need for heuristic tricks like gradient clipping.
By shifting the focus from "how many layers to stack" to "what equilibrium state to reach," this research direction provides a mathematically rigorous framework for building models that are effectively infinite in depth but finite (and constant) in memory.