Making a video classifier robust to adversarial attacks (tiny, deliberately crafted pixel changes that flip its prediction) normally means adversarial training. Adversarial training is a loop in which, at every update, the model first searches for the worst perturbation of each clip and then learns from it. For video this is very expensive, because each search step means a full forward and backward pass through a 3D network over many frames. The only systematic study of adversarial training for video worked on a single convolutional backbone and found that naive training wrecked clean accuracy, from 92% to 28% on UCF101 (Analysis and Extensions of Adversarial Training for Video Classification). Meanwhile, video practice has moved to frozen pretrained backbones with small trainable add-ons called adapters, and to a zoo of backbone families: convolutional networks, vision transformers, and hybrids. Nobody knows whether a cheap robustness add-on designed for one family transfers to the others. For temporal modules, this kind of transfer usually fails.
We propose a single robustness adapter that attaches to any frozen video backbone at an intermediate layer and is trained adversarially against perturbations of that layer's activations rather than of the pixels. Because the backbone below the adapter is frozen, its features can be computed once per clip and reused. Each step of the worst-case search then runs only through the layers above the adapter, which cuts the dominant cost of adversarial training by roughly the fraction of the network that sits below the insertion point. The open question is whether robustness learned against activation-space attacks holds up against real pixel-space attacks. We expect the answer to depend on the backbone family and on where the adapter sits, and measuring that dependence across families is the core contribution.
Core Claim
Our hypothesis is that, for frozen video backbones, most of the robustness attainable by parameter-efficient adversarial training can be had at a small fraction of the usual cost. Two conditions are needed: the adversary acts on a mid-network activation tensor inside a norm ball calibrated to how much pixel attacks actually move that tensor, and the adapter operates on a representation shared by all backbones. That shared representation is the layer output viewed as a time-by-tokens-by-channels tensor, where a convolutional feature map is simply flattened into tokens. We further predict that the best insertion depth differs systematically between convolutional and transformer backbones. Convolutions keep perturbations spatially local through early layers. Self-attention mixes them across space and time immediately, so we expect transformers to need deeper insertion.
To make the inner search concrete, let $h_k(x)$ be the frozen backbone's output at layer $k$ for clip $x$. Let $g_\phi$ be the adapter with parameters $\phi$, and let $f_{>k}$ be the frozen remainder of the network. We train
$$\min_{\phi} \sum_i \max_{\|\eta\|_\infty \le \rho_k} \mathcal{L}\big(f_{>k}(g_\phi(h_k(x_i) + \eta)), y_i\big),$$
where $\eta$ is the activation perturbation and $\rho_k$ is its budget. We set $\rho_k$ to a high percentile of $\|h_k(x+\delta) - h_k(x)\|_\infty$ measured over pixel-space attacks $\delta$ in the target $\epsilon$-ball, and we recompute it periodically on a small probe batch. This periodic recalibration is the only place where full-network pixel attacks are run.
Relation to Prior Work
Three video papers motivate the gap and all three pay full-model costs. Analysis and Extensions of Adversarial Training for Video Classification studies pixel-space adversarial training and a curriculum variant on one 3D ResNeXt-101. Its generative defense adds an entire 3D GAN, which raises cost rather than lowering it. We keep its attack recipe, including the step-size rule $\hat{\alpha} \approx 0.2(\epsilon+1)$, but freeze the backbone and move the search off the pixels. Improve Video Representation with Temporal Adversarial Augmentation uses adversarial noise as an attention regularizer, at a 15–25% training overhead. It depends on class activation maps and auxiliary batch normalization, which transformers lack, and it never tests white-box attacks. Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts searches adversarially over geometric transforms, not bounded noise, and retrains the full model.
Two lines of work supply our method. ST-Adapter shows that a bottleneck with a depthwise 3D convolution, inserted into a frozen image transformer, matches full fine-tuning on Kinetics-400 (82.0% versus 81.7%). We reuse its bottleneck shape, but its authors note it is untested on convolutional backbones, and it has never been attacked. TAMT gives similar evidence that frozen backbones with temporal tuning are cheap and strong. Robustness Tokens is the closest robustness precursor. It appends ten learnable tokens to a frozen image transformer and recovers DINOv2-Giant from 0.3% to 63.1% accuracy under attack in 200 steps. It is image-only, and token appending has no counterpart in a convolutional network, so it cannot answer the cross-family question.
One study constrains our expectations. Large-scale Robustness Analysis of Video Action Recognition Models found that convolutional video models benefit from perturbation training while transformers do not consistently do so. If this carries over to adversarial adapters, our method may help only one family. We treat that as a measurable outcome rather than a failure.
Research Program
We will build one adapter: a temporal depthwise convolution plus a single lightweight attention layer over the flattened tokens, wrapped in a residual connection and initialized to the identity. We will attach it at three relative depths (one quarter, one half, three quarters) to frozen backbones from three families. The convolutional family includes TSM, SlowFast, and X3D. The transformer family includes TimeSformer, MViT, and Video Swin. The hybrid family is Uniformer. Each backbone first receives a clean-trained task head so that the frozen starting point is competitive.
Training caches $h_k(x)$ for each clip. The inner loop runs multi-step projected gradient ascent only over $f_{>k}$. Every few hundred steps, a probe batch receives full pixel-space attacks, which are used to refresh $\rho_k$ and to log the gap between activation-space and pixel-space robustness. The analysis will map robust accuracy and wall-clock cost against insertion depth for every backbone. It will test whether the calibrated $\rho_k$ predicts the pixel-space robustness obtained. It will also check whether robustness concentrates in the adapter's temporal or spatial path, which we will determine by zeroing each path at test time.
Baselines
The naive reference is the frozen backbone with a clean-trained ST-Adapter-style adapter of identical size, evaluated under the same attacks. Its robust accuracy should be near zero, and it fixes the floor for clean accuracy and cost.
The strongest existing method is the curriculum adversarial training of Analysis and Extensions of Adversarial Training for Video Classification. On UCF101 with 3D ResNeXt-101, it reports 29.2% accuracy under an $\epsilon=8$ attack, against 23.3% for standard adversarial training, while keeping clean accuracy in the 80% range. We will rerun it on the same backbone with the same attack and data, and we will also run full-model curriculum training on one transformer, TimeSformer, to obtain a matched number. We found no reported result on Lacuna for adversarial training of video transformers. Beating this baseline means matching its robust accuracy within two points at no more than one third of its GPU-hours, or exceeding it at equal GPU-hours. In both cases clean accuracy must stay within three points of the naive reference.
The key ablation trains the same adapter, at the same location and with the same number of steps, using ordinary pixel-space attacks through the whole frozen backbone. This isolates the activation-space inner loop. If pixel-space adapter training is only marginally more robust but several times more expensive, the claim holds. If it is far more robust, then the activation-space shortcut is what fails, not the adapter.
For transformer backbones we will also extend Robustness Tokens to spatio-temporal token sequences. We will train it with its original two-part loss (match clean features, pull attacked features back to clean ones), with a matched step budget.
The input randomization defense of Mitigating Adversarial Effects Through Randomization will be applied frame-consistently. It is evaluated only under expectation-over-transformation attacks, meaning the attacker averages gradients over the random transforms. All methods share the same data splits, attack hyperparameters, and a tuning budget of eight configurations each.
Why not just…?
Why not just use single-step adversarial training on the adapter, or reuse perturbations across updates? Single-step attacks cut the number of search steps but still backpropagate through the entire frozen backbone at every step. Activation-space search cuts the cost of each step, so the two savings are complementary rather than competing. The real threat is that single-step training through the full network may simply be robust enough. We will run single-step pixel-space adapter training as an additional arm. If it matches our method at similar cost, we will report that and recommend combining the two.
Why not just extend Robustness Tokens to video transformers? For transformers this may genuinely work well, given its 63.1% recovery on images. It does not apply to convolutional networks, which Large-scale Robustness Analysis of Video Action Recognition Models suggests are the family most receptive to perturbation training. Its own authors also flag that the defense weakens if the tokens leak. Our transformer comparison will settle the within-family question. The cross-family question is where our adapter has a claim and the tokens have none.
Why not just add input randomization at inference, for free? It restored Inception-v3 from 1.1% to 98.2% under DeepFool in Mitigating Adversarial Effects Through Randomization. However, the research file notes it can be bypassed by attackers who average gradients over the transformations and that it is weak against broad single-step perturbations. Our evaluation runs exactly those adaptive attacks, so any apparent robustness that comes only from hiding gradients will be exposed.
Why not just fine-tune the full model with curriculum training, now that compute is cheap? It remains the accuracy ceiling, and we run it. However, it collapsed clean accuracy under standard settings in prior video work, and its cost scales with the number of search steps times the size of the full network. The cost-versus-robustness curve against this baseline is the main deliverable.
Proposed Experiments
Adversarial evaluation uses UCF101 for comparability with the prior video adversarial-training numbers, Something-Something-v2 because temporal order matters there, and Kinetics-400 for scale. Attacks are multi-frame $L_\infty$ projected gradient descent at $\epsilon \in \{2, 4, 8\}$ (pixel scale 0–255), with at least 20 steps and the cited step-size rule. Each attack is run both with and without awareness of the adapter, and with expectation-over-transformation for every randomized method. Kinetics400-P and UCF101-P measure whether adversarial robustness transfers to natural corruptions, reported as relative robustness so that results compare with the 0.88 (TimeSformer) and 0.91 (MViT) values in prior work. Cost is reported as GPU-hours and as the number of full-backbone backward passes.
Three checks are decisive. The first is whether activation-trained adapters retain at least 80% of the pixel-space robustness gained by the full-cost ablation on both families; if they do so on only one family, the cross-architecture claim narrows to that family. The second is whether the best insertion depth is deeper for transformers than for convolutional networks, as the mixing argument predicts. The third is whether the calibrated budget $\rho_k$ predicts pixel-space robust accuracy across depths, which would let practitioners choose the insertion point without running full attacks.
References
- Analysis and Extensions of Adversarial Training for Video Classification
- Improve Video Representation with Temporal Adversarial Augmentation
- Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning
- TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Large-scale Robustness Analysis of Video Action Recognition Models
- Mitigating Adversarial Effects Through Randomization