Calibrated Confidence as the Stopping Rule for Robust Equilibrium Networks

Some neural networks do not compute their answer in a fixed number of layers. A deep equilibrium model applies one layer repeatedly, as an iterative solver, until its hidden state stops changing, and it reads the prediction off that fixed point. These models save memory, but every prediction costs a variable and sometimes large number of solver steps. The cost gets worse after adversarial training, the standard defense in which the model is trained on inputs deliberately perturbed to fool it: robust equilibrium models need many more iterations and sometimes never converge. Adversarially trained models are also known to be systematically underconfident and to lose accuracy on ordinary inputs. We propose to train an equilibrium model so that its confidence at every intermediate solver step is calibrated, meaning a stated 80% confidence is right about 80% of the time, including on adversarial inputs. That calibrated confidence then serves as the rule for when to stop iterating.

The bet is that one training signal can address three problems that are usually handled separately. Calibrated intermediate confidence lets easy clean inputs exit after a few steps, which cuts solver latency. It keeps the model's uncertainty on hard or attacked inputs as an explicit signal that buys more computation, instead of training that signal away. And because the exit rule truncates the solver, the adversarial inner loop during training runs through a short trajectory, which reduces the cost of adversarial training. The idea succeeds only if the learned confidence ranks right and wrong intermediate predictions better than a model whose confidences are fixed after training. That condition is testable, and simple post-hoc rescaling cannot meet it.

Core Claim

A deep equilibrium model (DEQ) defines its features as the solution $z^*$ of $z^* = f_\theta(z^*, x)$. A solver such as Anderson or Broyden iteration produces a sequence $z_1, z_2, \dots$, and gradients are computed by implicit differentiation at the fixed point (Deep Equilibrium Models; Implicit Deep Learning). Every intermediate state has the same shape as $z^*$, so a single classification head $h$ can read a prediction from any step. Earlier work used this property to supervise random intermediate states during adversarial training (Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics), and also to defend by early exit or by averaging intermediate predictions (A Closer Look at the Adversarial Robustness of Deep Equilibrium Models). Neither work asks whether the confidence at step $k$ means anything. Our claim is that a DEQ trained with a calibration penalty on randomly sampled solver steps, applied to clean inputs, label-flipping adversarial inputs, and confidence-targeted adversarial inputs, yields intermediate confidences that work as a reliable anytime stopping rule. The resulting model should match a robust DEQ's clean and robust accuracy at substantially fewer average solver steps. It should also stay calibrated under attacks aimed at confidence, at a training cost close to intermediate-state adversarial training.

Concretely, at a randomly sampled step $k$ and on an input $x'$ that is either clean or adversarial, let $p_k = \mathrm{softmax}(h(z_k(x')))$, let $c_k = \max_j p_{k,j}$ be the confidence, and let $\hat y_k$ be the predicted label. We train

$$\mathcal{L}(\theta) = \mathbb{E}_{k,\,x'}\Big[\mathrm{CE}(p_k, y) + \lambda\,\big(c_k - \mathbf{1}[\hat y_k = y]\big)^2\Big],$$

where CE is cross-entropy, $y$ is the true label, and $\lambda$ weights the second term. That second term is a Brier-style penalty on whether the model's confidence matches its own correctness. It is differentiable and can be replaced by the kernel-density approximation of expected calibration error from KULeuven at LeQua 2022: Model Calibration in Quantification Learning. At test time the solver stops at the first step where $c_k \ge \tau$, or at a cap $K_{\max}$.

This design directly addresses a tension recorded in the literature. Yang et al. found that high-entropy intermediate states accompany unstable solver trajectories under attack, and they minimize entropy at test time with input entropy reduction (Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics). RobustBench, by contrast, documents that robust models are already underconfident and that calibration requires honest uncertainty (RobustBench: a standardized adversarial robustness benchmark). We read the high entropy on unstable trajectories as information. Rather than erase it, we calibrate it and then spend it: a low-confidence state means "keep iterating", not "push the input until the model is confident".

Research Program

The first component is the training recipe. We start from TRADES-style intermediate-state training on a DEQ, since Yang et al. show this is competitive with explicit ResNets. To it we add the calibration penalty above, evaluated at the same sampled steps. The adversarial inputs come from two cheap sources, both computed by projected gradient descent through a truncated solver trajectory rather than a converged one. The first source is the usual label-flipping perturbation. The second is a confidence-targeted perturbation from the maximum-miscalibration family defined in Calibration Attack: A Framework For Adversarial Attacks Targeting Calibration, which moves confidence without changing the label. Training on this second kind is what keeps the stopping rule from being trivially exploited. We deliberately avoid the mixed-integer and ADMM inner loop of ACCE-ACT (Towards Certification of Uncertainty Calibration under Adversarial Attacks), whose authors identify its overhead as the main practical barrier. Our method uses an empirical calibration adversary, so it carries no certificate. We say this plainly and treat ACCE-ACT as the reference for worst-case calibration, not something we claim to beat on certification.

The second component is the analysis of solver dynamics. For each trained model we log the per-example iteration count under the confidence rule and under the conventional residual-norm tolerance. We record the fraction of examples that never converge within $K_{\max}$, and how both quantities shift between clean and attacked inputs. The question is whether calibration training reduces or worsens the iteration inflation that robust DEQs show (A Closer Look at the Adversarial Robustness of Deep Equilibrium Models), and whether confidence predicts eventual correctness better than residual norm does.

The third component is adversarial evaluation of the exit rule itself. An anytime model creates two attack surfaces that a fixed-depth model lacks. An attacker can push some early step to be confidently wrong, so the model exits with a bad answer. An attacker can also keep confidence low at every step, a latency attack that forces $K_{\max}$ iterations. We build an exit-aware white-box attack that maximizes the confident-wrong margin over all steps up to $K_{\max}$, using the trajectory-ensemble gradients that Closer Look showed are necessary to avoid gradient obfuscation. In their audit, a DEQ that showed 78.03% robust accuracy under standard PGD had only 47.12% under the proper attack. We also build a latency attack that minimizes $\max_k c_k$. Because RobustBench excludes models with inference-time optimization loops, we cannot simply submit to its leaderboard. Every number will instead come from AutoAttack plus these trajectory-aware attacks, run against the full pipeline including the exit decision.

Baselines

The naive reference point is a TRADES-trained DEQ run to the solver tolerance, with no calibration loss and no early exit. It fixes what clean accuracy, robust accuracy, expected calibration error (ECE) and iteration counts look like when nothing is done. We will also report the same model with post-hoc temperature scaling, so that the naive case gets standard calibration practice.

The strongest robust-DEQ competitor is random intermediate state training under TRADES from Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics. It is the closest existing method: it already supervises intermediate states and already saves training compute by not running the solver to convergence. The authors report 51.50% "All-attack" robustness on CIFAR-10, against roughly 50.3% for a vanilla DEQ and 49.14% for their ResNet-18. They also report that input entropy reduction raises PGD-10 accuracy from 50.46% to 51.41%. We will retrain it with the same architecture, data augmentation, epochs and attack steps as our method, and we will give it the same exit rule with a tuned $\tau$. Beating it means matching its robust accuracy, measured with our trajectory-aware attacks rather than its original protocol, within one point. On top of that, our method must reach the same accuracy at clearly fewer mean solver steps, with lower ECE on clean and confidence-attacked inputs. To be thorough, we also run its test-time entropy reduction variant, because that is exactly the mechanism our design argues against.

The fixed early-exit and intermediate-ensemble defenses from A Closer Look at the Adversarial Robustness of Deep Equilibrium Models are the natural non-adaptive competitors on the latency axis. That paper reports 47.12% robustness for DEQ-Large against 53.58% for ResNet-18, and 51.70% for DEQ-XL against 56.60% for WRN-34-10. We run each at matched mean iteration count, so the comparison asks whether choosing the stopping step per example beats choosing it once for the whole dataset.

On worst-case calibration the reference is ACCE-ACT from Towards Certification of Uncertainty Calibration under Adversarial Attacks. It reports a reduction in certified calibration error on CIFAR-10 at radius 1.0 from 56.36% under standard adversarial training to 47.08%. We will compute its approximate certified calibration error on our models where that is tractable. We expect our method to cost much less to train but not to match its certificate. Empirical calibration-attack adversarial training (CAAT) and compression scaling from Calibration Attack: A Framework For Adversarial Attacks Targeting Calibration are the empirical calibration defenses we must beat on post-attack ECE. We found no reported result on Lacuna for any calibration metric on deep equilibrium models, so those comparisons will be newly run.

Two ablations isolate the key ingredient. The first removes the calibration adversary while keeping the calibration penalty on clean and label-flipping inputs. This tests whether robustness to confidence attacks is what makes the exit rule safe. The second keeps all of training but stops on residual norm instead of confidence. This tests whether calibrated confidence carries information beyond solver progress. A third, cross-architecture reference is AR-AT on WRN-34-10 from Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off, which reports 90.89% clean accuracy and 50.77% AutoAttack robustness at about 1.15× the cost of standard adversarial training. It shows how far clean accuracy can be pushed in explicit networks. We do not expect a DEQ to match it, and we treat it as context rather than a target.

All methods share the CIFAR-10 and CIFAR-100 training splits, perturbation radius, attack-step budget during training, and number of epochs. They also share a hyperparameter search budget, the same number of trials for $\lambda$, $\tau$ and the TRADES weight, selected on a held-out split containing both clean and adversarial examples. We report wall-clock training time on identical hardware.

Why not just…?

Why not just fit a temperature for each solver step after training and threshold that? This is the most serious threat, and we treat it as one. Temperature scaling preserves the ranking of confidences within a step. Thresholding a per-step temperature-scaled confidence is therefore equivalent to choosing a different threshold on raw confidence at each step, a policy we can tune directly. Per-step temperature scaling can fix average calibration, but it cannot improve how well confidence separates correct from incorrect predictions. That separation, not average calibration, determines the accuracy-versus-iterations curve of an exit rule. Our method helps only if in-training calibration changes the ranking. We measure this directly as the area under the ROC curve for predicting correctness from $c_k$ at each step, and as the full accuracy-versus-mean-iterations curve against a per-step tuned-threshold baseline. Temperature scaling is also reported to fail after calibration attacks (Calibration Attack: A Framework For Adversarial Attacks Targeting Calibration). If the per-step baseline matches our curves on clean and attacked data, the in-training component is unnecessary and we will report that.

Why not just use an explicit WideResNet with TRADES and temperature scaling, and skip the solver entirely? An explicit network has no solver latency to save, and on CIFAR-10 it is currently more robust per parameter (56.60% for WRN-34-10 against 51.70% for DEQ-XL, per A Closer Look at the Adversarial Robustness of Deep Equilibrium Models). Our contribution is not that DEQs beat ResNets. It is that a weight-tied iterative model, whose intermediate states share one classifier, can turn calibrated uncertainty into compute that adapts per input, which a fixed-depth network cannot do without extra exit heads. We include the explicit baseline so readers can see the absolute gap honestly, and we frame the result as reducing the latency and calibration penalties of implicit models.

Why not just use fast single-step adversarial training to cut overhead? Single-step training is known to overfit catastrophically, appearing robust to one-step attacks while failing multi-step ones. RobustBench's evaluation shows that defenses validated with weak attacks can overstate robustness by more than ten points (RobustBench: a standardized adversarial robustness benchmark). Our savings come from truncating the solver trajectory, not from weakening the attack. The two are orthogonal, and we will report a single-step variant of our recipe, evaluated with AutoAttack and trajectory-aware attacks, to show whether combining them is safe.

Why not just reuse joint inference and input optimization to remove the nested loop? Joint inference and input optimization in equilibrium networks matches PGD robust accuracy at about 3× the speed by solving for the hidden state, adjoint and perturbation together. The augmented system, however, needs many more solver iterations and a larger acceleration cache, and the paper never examines calibration. We will run it as a training-cost comparator. If it trains faster than our truncated recipe at equal robustness, it can supply the adversarial examples inside our objective, and the calibration and exit components still stand.

Proposed Experiments

The primary study is on CIFAR-10 with a DEQ of the size used by Yang et al. Every method is evaluated on five outputs. The first is clean accuracy. The second is robust accuracy under AutoAttack and under the trajectory-ensemble and exit-aware attacks. The third is ECE and Brier score on clean inputs and under the underconfidence, overconfidence and maximum-miscalibration attacks. The fourth is mean and worst-case solver iterations, including under the latency attack. The fifth is training wall-clock time. CIFAR-100 repeats the study with a larger label space, where calibration attacks were shown to raise ECE by more than tenfold without changing accuracy. Imagenette serves as a resolution and scale check.

The decisive checks come in four parts. First, at matched robust accuracy, the calibrated DEQ must reach the RIST-TRADES accuracy with substantially fewer mean solver steps, and its correctness ranking must beat the per-step tuned-threshold baseline. If neither holds, the idea fails. Second, under the exit-aware attack, accuracy must not fall below that of the same model run to convergence. If it does, the exit rule has become a new vulnerability, and the calibration-adversary ablation should reveal it. Third, under the latency attack, worst-case iteration counts must stay at or below those of the uncalibrated robust DEQ. Fourth, training time must stay close to RIST's, since the added calibration adversary reuses the same truncated gradient passes. Clean accuracy is a constraint throughout: because aggressive calibration penalties are known to cause underconfidence, we will sweep $\lambda$ and report the full trade-off curve rather than one favorable point.