One Feedback Pass: Top-Down Gain Control as a Cheap Robustness Adapter for Frozen Vision Transformers

Standard image classifiers read an image once, bottom to top. Early layers commit to edges and textures before the network has any idea what object it is looking at, and a small adversarial perturbation that corrupts those early features is carried forward unchecked. Brain-inspired networks that send high-level context back down to the early layers have repeatedly been reported to resist noise and attacks without expensive adversarial training. Nearly all of this evidence, however, comes from old convolutional backbones, many unrolled iterations, and weak single-step attacks. We propose testing the idea where it now matters: large, frozen Vision Transformers (image models that process an image as a sequence of patch tokens). We add exactly one top-down pass, in which the model's first-pass summary of the image multiplicatively rescales the features of its early layers before a second read. We train only this small feedback module, and we evaluate the whole two-pass system with an attacker who knows every parameter.

The opportunity lies in a gap between two literatures that do not talk to each other. Parameter-efficient defenses such as robustness tokens and dual classification tokens make frozen Transformers robust at low training cost, but they remain strictly feedforward, and one of them depends on its learned tokens staying secret. Feedback networks claim robustness that does not come from secrecy, but their claims have never survived a standardized adaptive evaluation. A single, fully differentiable feedback pass is the smallest design that can test whether top-down information is worth anything once capacity, compute, and attacker knowledge are held equal.

Core Claim

For a frozen Vision Transformer, a small module that maps the first-pass class representation to multiplicative per-channel gains on early-layer tokens, trained with a short adversarial objective on the module alone, will give better clean-versus-robust trade-offs than a feedforward adapter with the same number of parameters and the same training compute. We expect this advantage to hold under full white-box AutoAttack, including its gradient-free component. The claim is falsifiable in a specific way. If replacing the top-down signal with a constant learned gain, with a signal from another image, or with a same-layer lateral signal recovers the same robustness, then the feedback adds nothing beyond adapter capacity, and we will report that outcome.

Concretely, let $h$ be the frozen backbone's final class-token vector after a normal first pass on input $x$. Let $z_l$ be the token features entering early block $l$, and let $W_l$ be the feedback module's only weights. The second pass replaces $z_l$ with

$$\tilde z_l = z_l \odot \left(1 + \alpha \tanh(W_l h)\right),$$

where $\odot$ is channel-wise multiplication broadcast over tokens and $\alpha$ bounds how far feedback can rescale a channel. The multiplicative form follows Cognitive Steering in Deep Neural Networks via Long-Range Modulatory Feedback Connections, which showed on AlexNet that gain-style feedback amplifies task-relevant features without inventing absent ones. The output is the prediction of the second pass. Training minimizes cross-entropy on PGD-perturbed inputs attacked through both passes. It also includes a clean-consistency term that keeps the second-pass logits on clean images close to the frozen first-pass logits, which is how clean accuracy is protected.

How This Differs From What Exists

Two papers motivate the claim that feedback can buy robustness without full adversarial training. Brain-inspired predictive coding dynamics improve the robustness of deep neural networks froze a VGG16, trained feedback decoders only to reconstruct clean activations, and ran 19 recurrent steps at test time. Predify generalized this to EfficientNetB0 but reported that skip-connection architectures were hard to handle. Neural Networks with Recurrent Generative Feedback reached 48.90% versus 42.31% under PGD at $\epsilon=8/255$ on CIFAR-10, using iterative inference with a generative feedback model. Our work differs in three ways. It targets attention-based backbones with residual streams. It uses one pass instead of five to twenty. And it gives the feedback module an explicit adversarial objective instead of relying on emergent robustness from reconstruction.

Two other papers constrain the design. Obfuscated Gradients Give a False Sense of Security showed that unrolled iterative defenses often only appear robust because their gradients vanish or explode, and that such defenses collapse under approximate-gradient or black-box attacks. RobustBench warns about inference-time loops for the same reason. These warnings are why we use a single, short, differentiable feedback path that standard gradient attacks can traverse exactly, and why the gradient-free Square Attack is a required check.

The parameter-efficient feedforward defenses supply the setting and the baselines. Robustness Tokens trains 10 tokens on frozen DINOv2, DeiT-III, and OpenCLIP models in about 200 steps, but frames those tokens as a private key. Revisiting adapters with adversarial training uses dual classification tokens and an interpolation weight $\beta$ that must be chosen by hand. Our module is parameter-efficient in the same sense, but its effect depends on the input through the top-down signal. We evaluate it assuming the attacker has all of its weights. Reducing Excessive Margin explains why ordinary adversarial training costs clean accuracy: boundaries are pushed past the needed radius. We borrow its idea of anchoring to a clean teacher in a cheap form, with the frozen first pass serving as the teacher for the consistency term.

Research Program

The first stage builds the feedback module for ViT-B16 and DINOv2 backbones. Feedback targets one to four early blocks, gains are computed from the first-pass class token, and $\alpha$ is swept. Because the backbone is frozen, the trainable parameter count is a few thousand to a few hundred thousand, depending on whether $W_l$ is low-rank. We will match that count exactly in every feedforward control.

The second stage trains the module with a few-step PGD inner loop on ImageNet-1k, back-propagating through both frozen passes. This roughly doubles per-step cost relative to Robustness Tokens. We will report GPU-hours for every method so that "cheap" is measured, not asserted.

The third stage asks what the feedback actually does. We will measure whether second-pass early-layer features on attacked images move closer to their clean counterparts, using the cosine-similarity diagnostic of Robustness Tokens, which rose from 0.12 to 0.89 on DINOv2-G. We will also check whether learned gains vary by predicted class or stay nearly constant. If the gains are nearly constant, the module is functioning as a fixed adapter rather than as feedback.

A secondary question is whether the size of the feedback-induced change can stand in for the manual $\beta$ of dual tokens as an automatic clean-or-attacked signal. We treat this as analysis, not as a gate inside the defense, because a gate that switches feedback off would itself become an attack target.

Baselines

The naive reference is the frozen backbone with no defense. Under white-box attack, Robustness Tokens reports DINOv2-G accuracy falling to 0.3%, with clean accuracy of 85.6%. We will reproduce this at our own attack settings for each backbone. Any method that does not clearly exceed it under AutoAttack has failed.

The strongest existing competitors are the two frozen-backbone feedforward defenses. For Robustness Tokens, the reported figures are 85.2% clean and 63.1% under attack on DINOv2-G. For dual classification tokens on ViT-B16, Revisiting adapters with adversarial training reports 83.76% clean top-1 and 49.87% robust accuracy against strong $\ell_\infty$ attacks. The file describes that method's backbone treatment ambiguously, as both a shared co-trained backbone and a frozen one. We will therefore cite the reported number but rerun it in a strictly frozen-backbone setting for parity. Both baselines will be retrained by us on the same ImageNet data and backbones, with the same trainable-parameter count as our module and the same GPU-hour budget. Each gets an equal hyperparameter search. All three methods will be attacked with full knowledge of their learned parameters, which matters most for Robustness Tokens. Beating them means one of two outcomes under identical AutoAttack settings: higher robust accuracy at clean accuracy within half a point of the frozen model, or a Pareto curve over $\alpha$ (respectively $\beta$) that lies above theirs.

The ablations isolating the key ingredient are the heart of the comparison. We will train three variants with identical parameters, two forward passes, and identical training. In the first, the gain vector is a learned constant independent of $h$. In the second, $h$ is taken from a different, randomly paired image. In the third, the gain is computed from the same early layer it modulates, which is lateral rather than top-down. A further compute-matched control runs the backbone twice with Robustness Tokens inserted on the second pass. This separates "a second look" from "a second look guided by high-level context." The proposal's claim survives only if the true top-down variant beats all of these.

Why not just…?

Why not just add more robustness tokens until the parameter count matches? This is the most serious threat. Robustness Tokens already converge quickly and preserve clean accuracy, and a constant gain is mathematically close to a learned token's effect. Nothing in the file shows that feedforward adapters fall short at matched capacity. The cited limitations concern secrecy and capacity against large perturbation budgets. The constant-gain and token-count-matched controls in the Baselines section settle this directly. If they tie, we conclude that top-down feedback contributes nothing on frozen Transformers.

Why not just use Predify-style unsupervised feedback with no adversarial objective, which has zero adversarial training cost? That would be the cheapest option. However, the evidence for it is limited to VGG16 and EfficientNetB0 under iterative attacks without AutoAttack, and its own authors report difficulty with skip connections. Its robustness also grows over many unrolled steps, which is exactly the pattern that Obfuscated Gradients Give a False Sense of Security associates with masking. We will include a one-pass, reconstruction-trained variant of our module as a baseline. If it matches the adversarially trained one under Square Attack, the adversarial inner loop is unnecessary and we would drop it.

Why not just use fast single-step adversarial training on a small adapter? Single-step training is prone to catastrophic overfitting, and conventional adversarial training pays the excessive-margin tax on clean accuracy described in Reducing Excessive Margin. This option is also orthogonal to our question, since our inner loop can itself be shortened. We will therefore report every method, ours included, with both one-step and few-step inner loops.

Why not just ensemble intermediate exits instead of feeding back? Towards Adversarial Robustness with Early Exit Ensembles reports that its ensemble fails under the joint all-exit PGD-MAX attack unless paired with full adversarial training. Its results are also on biosignals, not on vision backbones. We include a parameter-matched early-exit ensemble on the frozen ViT, attacked jointly across exits, as an additional control.

Decisive Experiments

All robustness numbers will use the RobustBench AutoAttack ensemble on ImageNet at $\ell_\infty$ radius $4/255$: two APGD variants, FAB, and Square. Clean top-1 accuracy will be reported alongside, and ImageNet-C mCE plus ImageNet-A, -R, and -Sketch accuracy will test whether gains extend beyond the trained threat model. The ADE20K attacked mIoU protocol of Robustness Tokens, which rose from 4.7 to 23.3 there, will test whether early-layer gains help dense prediction.

The first decisive check is gradient integrity. Square Attack accuracy must not exceed APGD accuracy by more than a few points. A targeted adaptive attack that maximizes the loss and simultaneously pushes the gains toward zero or saturation must not break the defense. Failure on either test means the robustness is masking.

The second decisive check is the top-down ablation. The true feedback must beat the constant, mismatched-image, and lateral variants at matched compute and parameters.

The third check is cost. We will report training GPU-hours and inference latency against Robustness Tokens and dual tokens, and the method must stay at roughly twice single-pass inference cost.

If all three checks pass, the result is a usable defense whose robustness does not rely on secret parameters. If the ablation fails, we will have a clean negative result that the field currently lacks: at matched capacity on modern backbones, top-down feedback does not add robustness.