Deep equilibrium models are networks that do not stack layers. They apply one weight-tied layer over and over until the hidden state stops changing, and they output that resting point (the "fixed point"). They are attractive for memory-limited deployment, since one layer's weights stand in for an entire deep stack. Nobody, however, knows what happens when those weights are compressed to 2–4 bits.
In ordinary layered networks, rounding errors snowball from layer to layer. In an equilibrium model the same rounded layer is reused at every step, so its error re-enters the loop it came from. We argue this changes the problem in a useful way. If the layer is contractive, the quantized model does not accumulate error without bound. It settles at a displaced fixed point, and the size of the displacement is governed by a quantity the model already computes during training: the inverse of $(I - J)$, where $J$ is the layer's Jacobian at equilibrium.
This one observation yields three things. The first is a calibration objective for post-training quantization of implicit models. The second is a way to quantize provably stable equilibrium layers without breaking their stability guarantee. The third is a label-free, per-input signal of quantization damage, namely how many solver iterations the input needs. That signal lets the model spend higher precision only on inputs that need it, instead of giving every input the same bit budget.
Core Claim
Let $f_\theta(z, x)$ be the equilibrium layer, and let $z^*$ satisfy $z^* = f_\theta(z^*, x)$. Write $\tilde\theta$ for the quantized weights and $e(z) = f_{\tilde\theta}(z,x) - f_\theta(z,x)$ for the error the rounding injects at state $z$. If $f$ is $L$-Lipschitz in $z$ with $L<1$ and $\|e\|\le\varepsilon$, the quantized fixed point $\tilde z^*$ satisfies
$$\|\tilde z^* - z^*\| \le \frac{\varepsilon}{1-L}, \qquad \tilde z^* - z^* \approx (I - J_{f_\theta}(z^*))^{-1}\, e(z^*).$$
The bound follows from the contraction mapping theorem. The first-order form follows from the implicit function theorem, which is the same identity Deep Equilibrium Models uses for backpropagation. Three consequences follow, and each one is testable.
First, the error does not grow with the number of iterations the way it grows with depth in the 2-bit transformer stacks studied by Quantization Error Propagation. Instead the error is amplified by a per-input factor that blows up as the local contraction weakens.
Second, extra test-time compute cannot buy back lost precision. Running more solver iterations converges more tightly to $\tilde z^*$, not to $z^*$. This answers the question left open in the file's Gap 3: equilibrium models will not "think their way out" of coarse weights.
Third, the per-input amplification $1/(1-L(x))$ also sets how fast the solver converges. Inputs that need many iterations should therefore be exactly the inputs whose answers quantization damages most. If that holds, the iteration count, which is observed for free while running the cheap quantized model, is a difficulty estimator that needs no reward labels or verifiers. This removes the supervision cost that Learning How Hard to Think pays to train marginal-reward predictors.
The new contribution is treating the fixed point, not the layer output, as the object to be preserved under quantization, and using its geometry both to calibrate the weights and to decide per input how much precision to spend.
Research Program
Fixed-point-aware calibration. Layer-wise post-training quantization usually minimizes $\|W X - \tilde W X\|_F^2$ on calibration activations. We instead minimize the predicted displacement of the equilibrium:
$$\min_{\tilde\theta}\ \mathbb{E}_{x}\, \big\| (I - J_{f_\theta}(z^*))^{-1}\big(f_{\tilde\theta}(z^*,x) - f_\theta(z^*,x)\big)\big\|^2 .$$
The inverse-Jacobian product is applied with the same iterative linear solve the backward pass already uses, so the extra cost is a small multiple of one training step per calibration batch. A second variant evaluates the objective at the quantized fixed point $\tilde z^*$ and alternates between solving for $\tilde z^*$ and re-rounding. This is the recursive analogue of QEP's correction for upstream drift. Quantization Error Propagation supplies the idea of calibrating on corrupted rather than clean inputs. Its formulation needs distinct downstream layers to absorb corrections, and an equilibrium model has none, so the method has to be re-derived for the recursive case. QEP also sets its correction strength to a uniform hand-tuned value of 0.5. The Jacobian weighting replaces that hand tuning with a direction-wise weight that follows from the model's own dynamics.
Structure-preserving quantization of stable layers. Monotone operator equilibrium networks guarantees a unique fixed point by writing the weight matrix as $W=(1-m)I - A^\top A + B - B^\top$. Rounding $W$ directly can break this structure.
We instead round the factors $A$ and $B$ and rebuild $W$ from the rounded factors. Any rounded $\tilde A$ still gives a positive semidefinite $\tilde A^\top\tilde A$, and any rounded $\tilde B$ still gives an exactly skew-symmetric $\tilde B - \tilde B^\top$. Monotonicity, and therefore existence and uniqueness of the fixed point, survives at every bit width by construction.
For the $\ell_\infty$-contractive layers of Robust Implicit Networks via Non-Euclidean Contractions, we apply the analogous move: quantize in the parameterization that enforces the matrix-measure bound, then project if rounding pushes it over. This addresses the tension the file raises with the random rotations of QuIP. A rotation applied to the factors, $A \to AV$ with $V$ orthogonal, leaves $A^\top A$'s role intact up to a change of basis. That suggests outliers can be spread inside factor space without breaking monotonicity, and we test this directly.
Iteration count as a precision router. We run every input through the 2-bit operator first. Inputs whose solver needs more than a threshold number of iterations are re-solved with a 4-bit residual correction (the stored difference between 4-bit and 2-bit weights), warm-started from the 2-bit state. This reuses the observation in Deep Equilibrium Algorithmic Reasoning that solver convergence already allocates compute non-uniformly. Our change is to use that signal to allocate precision rather than iterations, and to calibrate the threshold on unlabeled data. Batching is handled by putting fallback inputs in a separate second-pass batch, which avoids the thread-divergence problem noted for adaptive-compute methods.
Baselines
Naive references. The floor is round-to-nearest quantization of the equilibrium layer's weights, solved with the same Broyden or Anderson solver and the same tolerance. The ceiling is the full-precision model. Every method runs from the same trained checkpoint, uses the same 128–512 calibration examples, and uses the same solver settings.
Strongest local post-training method. This is OPTQ/GPTQ-style local reconstruction, and QuIP's incoherence rotation plus adaptive rounding applied to the weight-tied layer at clean $z^*$. QuIP is the strongest existing low-bit method for large language models: it reports 2-bit WikiText-2 perplexity of 6.33 on Llama-2-70B, against OPTQ's 123.91. On equilibrium models, however, there is no reported result found on Lacuna for any quantization method. The comparison therefore has to be run, not quoted.
Strongest error-propagation method. This is QEP adapted to an unrolled equilibrium model. We unroll the solver for $K$ steps, treat it as a $K$-layer network with tied weights, and average QEP's per-step corrections into a single tied update, with its strength tuned on held-out data over the same grid we give our method. For context, QEP reports rescuing 2-bit Llama-2-7B from perplexity 17,783 to 11.97 when combined with QuIP. That number concerns explicit transformers and is not directly comparable.
Quantization-aware training. Retraining with simulated rounding is reported as an upper reference with its training cost. We do not treat it as a competitor at equal cost.
Winning criteria. Our calibration "wins" if, at 2 and 3 bits, it closes more of the gap to full precision than unrolled-QEP and QuIP at matched calibration compute. Our router "wins" if it matches uniform 4-bit accuracy while storing and computing mostly at 2 bits.
Ablations. The first removes the Jacobian weighting, which leaves plain local reconstruction at $z^*$. The second rounds $W$ rather than its factors. The third replaces iteration count with three alternative routing signals: random routing at the same fallback rate, output-entropy routing, and a probe trained on labels in the style of The LLM Already Knows.
Why not just…?
Why not just unroll the solver and apply QEP? This is the most serious competitor. If our approximation is right, unrolled tied-weight QEP is a truncated version of our objective as $K$ grows. It could match us when the contraction is strong and $K$ is large. Its cost grows with $K$, and it must collapse $K$ different corrections into one shared weight, which may be badly conditioned exactly where $L$ is near 1. The experiment sweeps $K$, and also sweeps the contraction margin by training models with different monotonicity parameters $m$. If unrolled QEP matches us at all margins and equal compute, the calibration contribution reduces to a cheaper implementation, and we will say so.
Why not just run more solver iterations at low precision? The fixed-point bound predicts this converges to the wrong point. We test it directly by plotting accuracy against iteration budget for 2-bit models with the tolerance tightened tenfold. If accuracy rises substantially, our central claim is wrong.
Why not just use 4 bits, where quantization is nearly free? Evaluating Quantized Large Language Models finds 4-bit within 2% of full precision for explicit models. We expect equilibrium models to be more fragile, because of the $1/(1-L)$ amplification. If they are not, uniform 4-bit is a fine answer, and the contribution narrows to the 2–3-bit regime and the router. Reporting where the precision cliff falls for implicit models is itself part of the result.
Why not just retrain with quantization in the loop? Training with simulated rounding may well recover most accuracy. It costs a full training run, though, and offers no stability guarantee unless combined with factor-space quantization. We report it as a reference point. The cost–accuracy comparison shows whether post-training quantization is worth having here.
Why not route with output confidence instead of iteration count? Softmax confidence is cheap, and it is the natural alternative signal. We compare the two by AUROC for predicting quantization-induced errors. Iteration count must beat entropy to justify the claim.
Proposed Experiments
The models are monDEQ on CIFAR-10, where the paper reports 74.0% at 172K parameters and 89.0% at 1M; the non-Euclidean contractive model on MNIST (97.72%); the DEQ-Transformer on WikiText-103 (perplexity 23.2); and DEAR on CLRS-30 (75.42%, out-of-distribution on 64-node instances). The last benchmark tests whether multi-step reasoning collapses first, as it does in quantized language models. Weights are quantized to 8, 4, 3, and 2 bits, with activations kept at 16 bits; joint weight-and-activation quantization is a stretch goal.
The measurements are: task accuracy or perplexity; measured fixed-point displacement $\|\tilde z^*-z^*\|$; the fraction of inputs on which the solver fails to reach tolerance; and iterations per input. For the router we add accuracy against average bits per input.
The first decisive check is whether measured displacement tracks the predicted $(I-J)^{-1}e$ and the $\varepsilon/(1-L)$ bound across trained contraction margins. The second is whether rounding $W$ directly causes solver non-convergence at 2 bits while factor-space rounding never does. The third is whether the Spearman correlation between per-input iteration count and per-input quantization damage is clearly positive on held-out data, and beats entropy. If the third check fails, the router is dropped and the proposal reduces to a calibration method for equilibrium models.
References
- Deep Equilibrium Models
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Monotone operator equilibrium networks
- Robust Implicit Networks via Non-Euclidean Contractions
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- Deep Equilibrium Algorithmic Reasoning
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Evaluating Quantized Large Language Models