Deep Equilibrium Models for Infinite Depth and Constant Memory

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2202 papers231 conceptsfrozen 2026-10-04

In traditional deep learning, we think of a model as a sequence of discrete steps. To make a model more powerful, we typically stack more layers. However, this "deeper is better" philosophy comes with a significant tax: memory. During training, backpropagation requires storing the intermediate activations of every single layer to compute gradients, leading to a memory footprint that scales linearly with depth.

Deep Equilibrium Models (DEQs), introduced by Shaojie Bai, J. Zico Kolter, and Vladlen Koltun, offer a radical alternative. Instead of passing data through a finite stack of layers, a DEQ defines a single, weight-tied layer and finds its fixed point.

The Intuition of Equilibrium

Imagine a standard Residual Network where every layer is identical. As you pass an input $x$ through more and more layers, the hidden state $z$ might eventually stop changing. Mathematically, the network reaches a state where:

$$z^* = f_\theta(z^*; x)$$

Here, $z^*$ is the equilibrium point. In a DEQ, we don't actually stack the layers. We simply declare that the output of the model is the solution to this fixed-point equation. This approach effectively simulates an infinite-depth network using only a single layer's worth of parameters.

This framing shifts the problem from "executing layers" to "root-finding." To find $z^*$, researchers use iterative solvers like Broyden’s method or Anderson acceleration. As Shaojie Bai demonstrated in Multiscale Deep Equilibrium Models, this can even be applied across multiple spatial resolutions simultaneously, allowing for highly efficient computer vision architectures.

Constant Memory via Implicit Differentiation

The most striking feature of DEQs is their memory efficiency. Because the output is defined implicitly by an equation, we don't need to store any intermediate states from the solver's iterations to compute gradients.

Using the Implicit Function Theorem, we can differentiate directly through the equilibrium point. The gradient depends only on the final state $z^*$ and the local derivative of $f$ at that point. This allows DEQs to achieve the representational power of massive networks while requiring constant memory during training—slashing usage by up to 88% compared to traditional models.

The Latency Trade-off

The primary "catch" with implicit models is computational time. While they save memory, they often require many iterations to converge to a stable fixed point.

As noted in GEQ: Gaussian Kernel Inspired Equilibrium Models by Mingjie Li and Yisen Wang, the iterative root-finding process is often more resource-intensive than a single forward pass of a standard CNN. This creates a fundamental trade-off: you trade wall-clock time for memory and parameter efficiency.

To mitigate this, Samy Wu Fung and colleagues proposed Jacobian-Free Backpropagation (JFB). JFB speeds up training by avoiding the expensive inversion of the Jacobian matrix, treating the gradient computation as a preconditioned descent direction instead. Similarly, Avik Pal and Alan Edelman introduced Continuous Deep Equilibrium Models, which treat DEQs as infinite-time Neural ODEs and use "skip" architectures to provide better initial guesses for the solvers, accelerating convergence.

Applications Across Domains

The robustness and efficiency of the equilibrium framework have led to its adoption in diverse fields:

  • Federated Learning: Long Tan Le and Choong Seon Hong developed FeDEQ, which uses compact DEQ representations to reduce communication overhead between edge devices and a central server.
  • Graph Neural Networks: Researchers like Guiduo Duan have used DEQs to unify label-inputted GNNs, creating memory-efficient models for node classification.
  • Inverse Problems: The LUSER framework by Peimeng Guan and Mark A. Davenport applies equilibrium regularizers to image deblurring and MRI reconstruction, achieving high accuracy with minimal memory.
  • Signal Processing: In massive MIMO systems, Yifan Ma and Jun Zhang used DEQs to compress channel state information, allowing mobile devices to dynamically trade off accuracy for battery life.

By moving away from the "layer-by-layer" paradigm, Deep Equilibrium Models provide a mathematically elegant way to scale neural networks to extreme depths while keeping the memory footprint firmly under control.

Papers in this version

Showing 60 of 202 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 137 papers
  2. Version 1 · 2026-10-04 · 202 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_solver_latency_418ed301,
  title = {iterative solver latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-solver-latency/rdr_418ed301906e4a08a992fc34960953fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_deep_equilibrium_models_for_infinite_dep_418ed301_v1,
  title = {Deep Equilibrium Models for Infinite Depth and Constant Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deep-equilibrium-models-for-infinite-depth-and-constant-memory/rdr_418ed301906e4a08a992fc34960953fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}