A weight-shared network (sometimes called a looped or recurrent network) applies the same block of weights several times in a row, so it can be as deep as a ResNet while storing only one layer's worth of parameters. Adversarially trained models, meaning models trained on worst-case perturbed inputs so that small crafted pixel changes cannot flip their predictions, are known to need very large networks. Small robust models lose substantially on both clean and attacked accuracy. That raises a basic question that no one has tested cleanly: is this "capacity tax" paid in parameters or in computation? If it is paid in computation, a weight-shared network should in principle be robust at a fraction of the storage cost. In practice it is not. Robust weight-shared models, including deep equilibrium models that iterate one block until its output stops changing, trail ordinary feedforward networks. We propose to find out whether that gap reflects missing capacity or failed optimization.
Our method trains an untied twin: an identical network in which every repetition of the shared block gets its own weights. We make this twin robust with a clean-accuracy-friendly adversarial training recipe. We then train the weight-shared student to reproduce the twin's hidden state after every repetition, on both clean and adversarial inputs. The twin is thrown away before deployment. The deployed model has the student's parameter count and the same compute as the twin, and no test-time optimization. If the student closes most of the gap to its twin, the capacity tax is largely a computation tax, and robust models can be made much smaller in storage. If it does not, we will have direct evidence that robust features need unique parameters per depth, which is a useful negative result.
Core Claim
We claim that most of the robust-accuracy and clean-accuracy gap between weight-shared and untied networks of equal compute comes from optimization, not capacity. We further claim it can be removed by matching the student's trajectory to an untied robust teacher, one repetition at a time, using supervision that exists only at training time.
The prior work that motivates this claim comes from deep equilibrium models. A Closer Look at the Adversarial Robustness of Deep Equilibrium Models shows that under correct white-box attacks such models lose to explicit networks: DEQ-Large (about 10M parameters) reaches 47.12% robust accuracy on CIFAR-10 against 53.58% for ResNet-18. That paper also shows that naive attacks overstate DEQ robustness badly (78.03% apparent versus 47.12% true). Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics finds that intermediate states become unstable under attack. Its remedy, Random Intermediate State Training (RIST), supervises random intermediate states with the model's own label loss. It then adds a test-time input-optimization step, IER, to reach 51.50% on its all-attack evaluation. Two things separate our proposal from RIST. We supervise every intermediate state against an external target, namely the untied twin's representation at the same depth, rather than only against the class label. And we add nothing at inference.
On the parameter side, Deep Anchored Convolutional Neural Networks supplies the architecture: a shared convolutional core with per-repetition batch normalization and small regulator modules that matches ResNet-34 on clean data. Tied-LoRA supports the same design principle, one shared matrix plus tiny per-layer scaling vectors, for fine-tuning. Neither was ever trained or evaluated adversarially.
Research Program
We build a family of weight-shared image classifiers: a stem, a single residual block applied $T$ times with per-repetition batch-norm and per-channel scales, and a head. We unroll a fixed number of repetitions ($T \in \{4, 8, 16\}$) rather than solving for a fixed point. Unrolling gives exact gradients, which sidesteps the solver-induced gradient masking documented for deep equilibrium models. For each student we build its untied twin, the same graph with $T$ independent blocks, which has roughly $T$ times the block parameters at identical FLOPs.
We train the twin with Helper-based Adversarial Training (HAT) from Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off. HAT adds "helper" examples extrapolated twice as far as the attack and labeled by a standard-trained model, which keeps decision margins from growing unnecessarily and recovers about 2 points of clean accuracy over TRADES at matched robustness. We use the public imrahulr/hat codebase. The student is then trained with the same HAT objective plus a trajectory-matching term:
$$ \mathcal{L} = \mathcal{L}_{\text{HAT}}(\theta_s) + \lambda \sum_{t=1}^{T} \big\| h^s_t(\tilde x) - h^u_t(\tilde x) \big\|_2^2 , $$
Here $h^s_t$ and $h^u_t$ are the student's and frozen twin's hidden states after repetition $t$, and $\tilde x$ ranges over the clean input, the student's own adversarial example, and the helper example. $\lambda$ is a single weight. Because the two networks share every shape, no projection heads are needed. The added training cost is one frozen forward pass of the twin per batch plus the twin's own training, and we will report both in wall-clock time.
The analysis half of the program tests where the gap lives. HAT measures excessive margin along the initial adversarial direction at the output. We will measure it at every repetition $t$, by attaching a linear probe to $h_t$ (probes discarded after analysis). This asks whether weight-shared models inflate margins early in the trajectory, which is the open question in the file's Gap 3. We will also track how the student's input-output sensitivity (an empirical Lipschitz estimate) grows with $t$. Expressive Power of Implicit Models shows this quantity rising from about 0.1 to 5.0 over iterations in deblurring. That result suggests shared blocks build sharp functions by repetition, which is precisely the mechanism our claim needs.
Baselines
The naive reference point is the weight-shared student trained with plain TRADES or HAT and no teacher, at the same $T$, data, epochs and attack budget. Every gain we report is measured against it.
The strongest existing method for adversarially training weight-tied models is RIST from Improving Adversarial Robustness of Deep Equilibrium Models. That paper reports 51.50% all-attack robust accuracy on CIFAR-10 with RIST plus IER under TRADES, against roughly 50.3% for its vanilla DEQ. We will reimplement RIST on our unrolled students with the same epochs and a matched hyperparameter sweep, and report it both with and without IER. The fair comparison is RIST without IER, since our model adds no test-time cost. Beating it means higher AutoAttack accuracy at equal or higher clean accuracy, with a margin larger than the seed-to-seed spread over three seeds.
The strongest distillation competitor is multi-teacher adversarial distillation (MTARD) from Enhanced Accuracy and Robustness via Multi-teacher Adversarial Distillation. It distills clean and adversarial WideResNet teachers into a compact student at the output level and reports 89.26% clean accuracy for MobileNet-V2 on CIFAR-10. We will run MTARD with the untied twin as its adversarial teacher, and with a WideResNet-34-10 teacher, into the same weight-shared student.
The key ablations isolate the ingredients. First, output-only distillation from the twin, which removes the per-repetition term. Second, trajectory matching on clean inputs only. Third, an initialization-only variant: set the shared block to the average of the twin's blocks, then fine-tune with HAT and no matching loss.
The ceiling is the untied twin itself. As parameter-matched explicit references we use PreAct ResNet-18 with HAT, which the HAT paper reports at about 49% AutoAttack, and a ResNet of the same parameter count as the student. The student is judged by the fraction of the twin's clean and robust accuracy it retains per stored parameter.
Why not just…?
Why not just run HAT or TRADES on the weight-shared network with more repetitions? Adding repetitions is free in parameters, so if the gap were purely compute it should shrink with $T$ under plain training. The DEQ evidence argues otherwise. Running a single block to equilibrium, effectively unlimited repetitions, still leaves DEQ-Large about six points behind ResNet-18 in A Closer Look at the Adversarial Robustness of Deep Equilibrium Models. Our $T$-sweep of the naive baseline measures this directly. If plain HAT's gap to the twin closes as $T$ grows, our teacher is unnecessary and we will say so.
Why not just distill the outputs of a big robust teacher, as MTARD does? This is the most serious threat, because output distillation is simpler and already reduces the capacity tax for compact models. Our bet is that a weight-shared block receives conflicting gradients from different depths, and that telling it what each repetition should compute resolves the conflict in a way a single output target cannot. The output-only-from-twin ablation settles this. If it matches full trajectory matching within noise, the contribution collapses to "use an untied twin as teacher," and we will report that narrower finding.
Why not just train untied and then average the weights into one block? Averaging is nearly free and could supply a good starting point. However, the repetitions of a trained untied network need not compute similar functions, and averaging ignores the order of the trajectory. The initialization-only ablation tests this directly and is also combined with our loss.
Why not just deploy the untied twin, or use test-time defenses like IER? The twin stores roughly $T$ times the block parameters, which is exactly the cost we are trying to avoid. IER requires gradient steps on every query, as its own paper concedes. Both are reported, as ceiling and as with-overhead reference, but neither meets the deployment constraint.
Proposed Experiments
We train on CIFAR-10, CIFAR-100 and SVHN under $\ell_\infty$ with $\epsilon = 8/255$. We scale the best configuration to ImageNet-100, which the HAT paper already uses. We report clean accuracy, AutoAttack accuracy (from Reliable evaluation of adversarial robustness, in RobustBench's protocol from RobustBench), stored parameters, FLOPs and training wall-clock. All results use three seeds, at $T \in \{4, 8, 16\}$.
Because the student is trained to imitate a known network, evaluation must include attacks built against that network. We add transfer attacks crafted on the twin, the black-box Square attack, and PGD with hundreds of steps and restarts. Any gap between gradient attacks and these attacks is treated as evidence of gradient masking, not robustness.
The decisive check is a single curve per dataset, plotting the fraction of the twin's robust and clean accuracy retained by the student against $T$. The hypothesis predicts that trajectory matching keeps this fraction near one as $T$ grows, while the naive and output-only students fall away. The hypothesis is refuted if trajectory matching tracks output distillation, or if the retained fraction decays with $T$ at the same rate as the naive baseline. That outcome would indicate robust features really do need unique parameters per depth.
The secondary check is the per-repetition margin analysis. If the naive student shows inflated margins at early repetitions that trajectory matching removes, that finding links the clean-accuracy loss in shared networks to HAT's excessive-margin account and extends it from the output into the network's depth.
References
- A Closer Look at the Adversarial Robustness of Deep Equilibrium Models
- Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics
- Deep Anchored Convolutional Neural Networks
- Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying
- Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off
- Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scaling
- Enhanced Accuracy and Robustness via Multi-teacher Adversarial Distillation
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- RobustBench: a standardized adversarial robustness benchmark