A deep equilibrium model is a network that, instead of stacking many layers, repeatedly applies one shared layer until its hidden state stops changing, and then reads out a prediction from that resting point. Training is cheap in memory because gradients are computed directly at the resting point using the implicit function theorem, a rule that says how a solution moves when the equation defining it changes. A Laplace approximation is a cheap way to turn a trained network into a Bayesian one. It fits a Gaussian over the weights around their trained values, with its width set by the curvature of the loss, and from that Gaussian it produces error bars on each prediction. These two ideas have never been joined. Current Laplace tools assume a layer-by-layer network with stored activations, and an equilibrium model throws those activations away.
This note proposes a post-hoc Laplace approximation whose curvature and predictive variance are computed through the fixed point with the same linear solve the model already uses for its gradients. The method needs memory that stays constant in the number of solver iterations, and it never resamples the equilibrium layer, so it cannot make the solver diverge. Its central claim goes beyond "Bayesian DEQs now exist." Implicit differentiation contains the factor $(I - \partial f/\partial z)^{-1}$, which becomes large when the fixed point is close to unstable. That factor carries over into the predictive variance, so the model's stated uncertainty should automatically rise on inputs where its own equilibrium is fragile. Uncertainty methods applied only to the last layer cannot see this effect. The proposal tests whether the effect exists, whether it is useful, and whether a free heuristic such as solver iteration count already captures it.
Core Claim
Let $f(z, x; \theta)$ be the equilibrium layer, $z^*(x)$ its fixed point, $h(z^*; \phi)$ the readout, and $J_f = \partial f/\partial z$ evaluated at $z^*$. The Jacobian of the network output with respect to the equilibrium-layer weights $\theta$ is
$$ \mathcal{J}_\theta(x) = \frac{\partial h}{\partial z^*}\,(I - J_f)^{-1}\,\frac{\partial f}{\partial \theta}. $$
The linearized Laplace approximation uses this Jacobian twice. It forms a Generalized Gauss–Newton curvature $G = \sum_n \mathcal{J}_\theta(x_n)^\top \Lambda_n \mathcal{J}_\theta(x_n) + \delta I$, where $\Lambda_n$ is the Hessian of the loss with respect to the logits and $\delta$ is the prior precision. It then gives a predictive logit covariance $\Sigma(x) = \mathcal{J}_\theta(x)\, G^{-1}\, \mathcal{J}_\theta(x)^\top$, combined with the usual last-layer term for $\phi$. Every product with $\mathcal{J}_\theta$ or its transpose costs one linear solve against $I - J_f$, which is exactly the solve in the DEQ backward pass. The whole procedure therefore inherits the constant-memory property.
The scientific claim concerns that inverse. If $f$ is contractive with $\|J_f\| \le \gamma < 1$, the Neumann-series bound gives $\|(I-J_f)^{-1}\| \le 1/(1-\gamma)$. Predictive variance can therefore grow like $1/(1-\gamma(x))^2$ as the local contraction rate $\gamma(x)$ approaches one. We hypothesize that corrupted and out-of-distribution inputs push $\gamma(x)$ toward one. If so, full-block linearized Laplace will separate in-distribution from shifted inputs better than last-layer Laplace on the same backbone. The improvement should disappear when the inverse is replaced by the identity.
Where This Sits
The proposal takes its method from two papers. Laplace Redux builds post-hoc Laplace along four choices: which weights get a distribution, how curvature is approximated, how the prior is tuned by marginal likelihood, and how the predictive is integrated. It shows last-layer variants are almost free. All of its curvature backends assume explicit layers with cached activations. We keep its choices but replace the curvature engine with implicit-differentiation products. General adjoint-differentiated Laplace approximation differentiates the optimality condition of a Laplace approximation, but only in shallow latent Gaussian models. Our use of the implicit function theorem is different: we differentiate a network's forward equilibrium, not the posterior mode.
Two lines of work constrain the design. Stabilizing Equilibrium Models by Jacobian Regularization shows that solvers diverge when the spectral radius of $J_f$ reaches one, and it penalizes $\|J_f\|_F$ to prevent this. This rules out naive weight sampling, because posterior samples of $\theta$ can cross the stability boundary. It also creates a direct tension with our claim: regularization shrinks exactly the amplification factor we expect to carry signal. Improving Bayesian Neural Networks by Adversarial Sampling documents that random weight samples can fall well below the mean network in accuracy. The linearized predictive sidesteps that problem because it never evaluates a perturbed network.
On Training Implicit Models supplies a cheaper approximation to the inverse. Its phantom gradients replace exact inversion with truncated Neumann or unrolled updates. We reuse these as a fast, biased curvature engine and measure what the truncation costs in uncertainty quality.
The closest connecting work is DDEQs. It places distributions over inputs (point clouds treated as measures) but keeps weights deterministic, so it addresses a different question. BARNN places variational distributions on recurrent weights but unrolls explicitly. Its reported failure of MC dropout on long rollouts (ECE 0.16–0.30 against 0.05) motivates avoiding per-iteration stochasticity inside an iterated map.
Research Program
The first component is the curvature engine. We will implement GGN-vector products for a trained DEQ. Each product does a forward-mode pass through $\partial f/\partial\theta$, a solve against $I-J_f$ using the model's own backward solver, the readout Hessian, and then the transposed path. From these products we build a low-rank-plus-diagonal posterior: Lanczos on $G$ gives the top $k$ eigendirections, and a Hutchinson-style diagonal estimate covers the rest. This choice is deliberate. Kronecker factorization assumes independent layers, and the inverse in $\mathcal{J}_\theta$ couples every weight in the block.
We will tune the prior precision by maximizing the Laplace marginal likelihood, as in Laplace Redux. At test time we need $\mathcal{J}_\theta(x)$ applied to the $k$ eigenvectors, which is $k$ solves sharing one matrix and can be run as one batched multi-right-hand-side solve. We will report the exact memory and wall-clock overhead relative to a single deterministic forward pass.
The second component is analysis. We will record, per test input, the solver residual, the iteration count, a power-iteration estimate of $\rho(J_f)$, and the predictive variance. With these we can check whether variance tracks $1/(1-\rho)^2$, and whether that relationship survives once the cheap solver statistics are accounted for.
The third component is the interaction with stabilization. We will train backbones at several Jacobian-regularization strengths and measure how in-distribution accuracy, calibration, and shift detection trade off as the amplification is suppressed.
All experiments use DEQ image classifiers trained on CIFAR-10 and CIFAR-100. They are evaluated on clean test sets, on CIFAR-10-C across all corruption types and severities, and on cross-dataset detection, with CIFAR-100 treated as out-of-distribution for CIFAR-10 models. Metrics are accuracy, negative log-likelihood, expected calibration error, AUROC for out-of-distribution detection, peak memory, and latency.
Baselines
The naive reference is the deterministic DEQ's softmax with temperature scaling fitted on held-out data, trained as in On Training Implicit Models. That paper reports 93.8% CIFAR-10 accuracy with exact implicit gradients and 95.0% with phantom gradients. We use the phantom-gradient-trained backbone, so our accuracy target is 95.0%. The linearized predictive should leave the argmax essentially unchanged; it must not fall below this figure by more than noise while improving NLL and ECE.
The strongest cheap competitor is last-layer Laplace from Laplace Redux. It is applied to the same frozen backbone, uses the same marginal-likelihood prior tuning, and is run with the laplace library on the readout head. It is the most dangerous baseline because it costs almost nothing and is often sufficient. No reported result was found on Lacuna for last-layer Laplace on a DEQ or on CIFAR-10-C. Laplace Redux states only qualitatively that overconfidence drops on corrupted data, so we will produce this number ourselves.
The strongest expensive competitor is a deep ensemble of five independently trained DEQs. We compare it at equal test-time compute, by counting solver iterations, and also unconstrained, so that its accuracy advantage is visible.
As a reference point from the probabilistic-ensembling literature, UraeNAS reports gains over deterministic DrNAS of 0.86% accuracy and a 42% ECE reduction on CIFAR-10, and 2.43% accuracy and a 30% ECE reduction on CIFAR-10-C. These are relative improvements on a different backbone, so they are a target scale, not a like-for-like number.
A sampled Laplace variant draws $\theta$ from the same Gaussian and re-solves the fixed point. It shows what the linearization avoids, and we report its divergence rate. A Monte Carlo dropout DEQ holds one mask fixed for each full solve, which avoids the moving-target problem, and serves as the standard stochastic-weight alternative.
The ablation that isolates the key ingredient keeps everything identical but replaces $(I-J_f)^{-1}$ with the identity, as if the block were applied once. A second variant uses a $k$-term Neumann truncation. If the identity ablation matches full linearized Laplace, the equilibrium-amplification claim is false.
All methods share the same trained backbones, the same validation split for prior and temperature tuning, and an equal hyperparameter search budget. Beating last-layer Laplace means a statistically significant gain in CIFAR-10-C NLL and in out-of-distribution AUROC across three seeds, with memory overhead independent of solver iteration count.
Why not just…?
Why not just use last-layer Laplace on the equilibrium features? It is cheap and well established, and it may well be enough on clean data. Its blind spot is structural. It treats $z^*$ as fixed, so it cannot register that the equilibrium itself shifted or became fragile under corruption. The identity-inverse ablation sits between last-layer Laplace and our method, so the comparison across the three tells us whether any gain comes from weight uncertainty in the block or specifically from propagating it through the fixed point. If last-layer Laplace matches us on CIFAR-10-C, we will report that the block posterior adds nothing.
Why not just use solver iteration count or residual as the uncertainty signal? This is the most serious threat. These quantities are free, and they plausibly correlate with $\rho(J_f)$ for the same reason our variance does. We will evaluate them as standalone detectors and in a combined detector. If they match our AUROC, the honest conclusion is that the Bayesian machinery recovers a known heuristic. Our method would then remain useful only where calibrated probabilities matter, which NLL and ECE measure and an iteration count does not provide. We will state whichever outcome occurs.
Why not just ensemble several DEQs? Ensembles likely win on accuracy. However, DEQ forward passes are already reported to be two to four times slower than explicit models of similar size (Deep Equilibrium Models), so an ensemble multiplies training cost by the number of members and multiplies root-finding at test time. Our claim is a better calibration-per-solve trade-off from one trained model, and the equal-compute comparison settles it.
Why not just sample weights and re-solve, as in ordinary Bayesian networks? Samples that push $\rho(J_f)$ past one make the solver diverge (Stabilizing Equilibrium Models by Jacobian Regularization). Surviving samples can also degrade accuracy (Improving Bayesian Neural Networks by Adversarial Sampling). We measure both effects directly in the sampled-Laplace baseline instead of assuming them.
Decisive Checks
The idea stands or falls on three results.
The first is the identity-inverse ablation. On CIFAR-10-C NLL and on CIFAR-100 out-of-distribution AUROC, full linearized Laplace must beat the ablation, or the amplification claim is refuted.
The second is the iteration-count comparison. Predictive variance must carry detection signal beyond the cheap solver statistics, which we test with a combined detector, and must give lower ECE than temperature scaling plus an iteration-count heuristic.
The third is the regularization sweep. If the claim is right, increasing Jacobian regularization should monotonically shrink the gap between our method and last-layer Laplace. That would be the first measured trade-off between solver stability and the quality of a DEQ's uncertainty.
Throughout, we report peak memory as solver iterations scale, to confirm that the uncertainty estimate costs constant memory in effective depth, as the gradients do.
References
- Laplace Redux - Effortless Bayesian Deep Learning
- General adjoint-differentiated Laplace approximation
- Stabilizing Equilibrium Models by Jacobian Regularization
- Improving Bayesian Neural Networks by Adversarial Sampling
- On Training Implicit Models
- DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flows
- BARNN: A Bayesian Autoregressive and Recurrent Neural Network
- Unified Probabilistic Neural Architecture and Weight Ensembling Improves Model Robustness
- Deep Equilibrium Models