Adversarial training is the standard way to make a classifier robust to worst-case input perturbations. The model is trained on inputs that an inner attack loop has deliberately corrupted. On video this works badly on two counts. It is expensive, because every training step must run several forward and backward passes over a full multi-frame clip to build the attack. It is also destructive: on UCF101, standard adversarial training of a 3D ResNeXt-101 dropped clean accuracy from 92% to 28% (Analysis and Extensions of Adversarial Training for Video Classification). This note proposes that both problems share one cause. The training-time attacker is allowed to change its perturbation freely from frame to frame, so it injects fake motion. To resist it, the model learns to discount real differences between frames, and those differences are exactly the motion cues that action recognition depends on.
The proposal is to limit how fast the training-time attacker's perturbation may change over time. The perturbation is expressed through a few slowly varying temporal basis functions, and the cutoff on temporal frequency, rather than the usual pixel budget $\epsilon$, becomes the knob that trades robustness against clean accuracy. A slow perturbation lives in a small space and stays useful across training epochs. We can therefore store it per video and refresh it with one or two attack steps instead of seven to ten. We test this in two regimes. The first is full-parameter training of a 3D CNN, where the clean-accuracy collapse is documented. The second is a cheap regime in which a frozen video transformer is hardened only through a handful of learnable input tokens. The central falsifiable bet is that robustness learned against band-limited attacks holds up against unrestricted attacks at the same $\epsilon$. If it does not, the idea fails, and the experiments are built to show that cleanly.
Core Claim
Write a clip perturbation as $\delta \in \mathbb{R}^{T \times H \times W \times C}$ for $T$ frames. Let $P_{>\omega}$ be the projection onto temporal discrete-cosine frequencies above a cutoff $\omega$, taken independently at each pixel and channel. The training objective is
$$\min_{\theta} \; \mathbb{E}_{(x,y)} \max_{\|\delta\|_\infty \le \epsilon,\; P_{>\omega}\delta = 0} \mathcal{L}\big(f_\theta(x+\delta), y\big),$$
where $f_\theta$ is the video model, $\theta$ are the trainable parameters (all weights, or only robustness tokens), and $\mathcal{L}$ is the training loss. Setting $\omega = 0$ gives a static perturbation stamped identically on every frame, the regime that Adversarial Framing for Image and Video Classification and Universal 3-Dimensional Perturbations for Black-Box Attacks on Video Recognition Systems show is already a strong attack. Setting $\omega = T-1$ recovers ordinary unrestricted PGD (projected gradient descent, the standard iterative attack).
We parameterize the perturbation as $\delta = \sum_{k=0}^{\omega} c_k\, b_k(t)$, with $b_k$ the temporal cosine basis and $c_k \in \mathbb{R}^{H\times W\times C}$. The $\ell_\infty$ constraint does not project exactly in this basis, so we alternate between clipping and re-projection and report how often the result falls outside the budget.
The claim has three parts, each testable. First, at matched robustness to unrestricted attacks, models trained against small $\omega$ lose less clean accuracy than models trained against full-band attacks. The loss they avoid should be concentrated on temporally demanding data such as Something-Something v2, not on UCF101. Second, because low-band perturbations are nearly universal across frames, they can be persisted per video and refreshed cheaply, which cuts attack cost several-fold. Third, an $\omega$ schedule beats the $\epsilon$ schedule of curriculum adversarial training at equal compute.
The direction of the first part is supported by Large-scale Robustness Analysis of Video Action Recognition Models. That study found that temporal perturbations collapse accuracy on SSv2 but barely affect UCF101 or Kinetics. In other words, the motion features at stake matter mostly on order-sensitive data. That is where a hardening procedure that teaches order-insensitivity should do visible damage.
How This Relates to Prior Work
Two papers supply the starting point. Analysis and Extensions of Adversarial Training for Video Classification ran full PGD adversarial training on a 3D ResNeXt-101. It found that training against smaller-than-optimal budgets gave both better robustness and better clean accuracy, and it proposed two fixes. Curriculum adversarial training (CAT) grows $\epsilon$ over epochs. Generative adversarial training (GAT) adds a 3D denoising GAN in front of the classifier. We keep that paper's empirical lesson that a weaker training attacker can be better, but we move the weakening from the pixel budget to the time axis. We also avoid GAT's added generator and discriminator.
Robustness Tokens: Towards Adversarial Robustness of Transformers freezes a vision transformer and trains about ten appended tokens. It uses two losses: one keeps clean features unchanged and one pulls attacked features back to the clean ones. With DINOv2-G it kept clean ImageNet accuracy at 85.2% against 85.6% and recovered white-box accuracy from 0.3% to 63.1% in 200 steps. We reuse this method as our cheap regime on TimeSformer and MViT. One point about it matters for cost. Robustness tokens remove the cost of updating parameters, but not the cost of generating the attack, because input gradients still flow through the whole frozen backbone. On video, attack generation is the dominant cost, and band-limiting plus persistence is aimed precisely at it.
The temporal-consistency defenses motivate the frequency view but play a different role. AdvIT warps past frames with optical flow and flags frames whose predictions disagree. Identifying and Resisting Adversarial Videos Using Temporal Consistency counts label changes between frames and then repairs frames by motion-compensated patching or by ComDefend compression. Both work at test time, both rely on flicker being an anomaly, and neither hardens the model. Two other papers constrain how far such defenses can go. Towards Understanding Adversarial Robustness of Optical Flow Networks shows that flow estimators are themselves attackable: a patch raised FlowNetC's end-point error from 11.50 to 52.66. The attacks in Structured Universal Adversarial Attacks on Object Detection for Video Sequences are deliberately static to evade temporal filters.
Read together, these papers suggest a division of labor. Temporal checks are naturally suited to the high band, so model hardening should concentrate on the low band, which those checks cannot see. Our main threat model, however, is a white-box attacker on the classifier alone. Under that threat model the high band must still be covered, and the experiments test whether it is.
Research Program
We first establish whether the clean-accuracy problem exists in each regime. In the full-parameter regime we reproduce standard adversarial training of 3D ResNeXt-101 on UCF101, using the step-size rule $\hat{\alpha} \approx 0.2(\epsilon+1)$ reported by Analysis and Extensions of Adversarial Training for Video Classification. In the token regime we port robustness tokens unchanged to Kinetics-400-pretrained TimeSformer and MViT, trained against full-band PGD, and fine-tuned for UCF101 and SSv2.
This step can reshape the project. If tokens trained against full-band attacks already lose under one point of clean accuracy on both datasets, the accuracy half of the claim is moot in the cheap regime, as the image results for robustness tokens hint it might be. The proposal would then rest on cost in that regime and on accuracy in the full-parameter regime only. We will report that outcome as a result rather than bury it.
We then sweep the cutoff $\omega \in \{0, 1, 2, 4, T-1\}$ for $T=16$-frame clips at fixed $\epsilon$, in both regimes. For each setting we measure clean accuracy and robust accuracy against unrestricted PGD, against static framing-style attacks, and against a flicker-only attack confined to the high band. We also measure order sensitivity: the drop in SSv2 accuracy when a clip is reversed or shuffled. A model that has learned to ignore motion will show a smaller drop.
The cost mechanism is a per-video store of the coefficients $c_k$. The store is warm-started from the previous epoch and refreshed with $s \in \{1, 2\}$ attack steps. Cost is counted in backbone forward-backward passes and in GPU-hours. Finally, we compare an $\omega$ curriculum, which widens the temporal band over training, against CAT's $\epsilon$ curriculum at equal passes.
Baselines
The naive reference point is the undefended model: clean fine-tuned TimeSformer, MViT, or 3D ResNeXt-101 with no hardening. It fixes the clean ceiling and the floor under attack. Analysis and Extensions of Adversarial Training for Video Classification reports this floor as 7.0% under masked PGD for the 3D CNN. We found no reported adversarial result on Lacuna for TimeSformer or MViT, so we measure those floors ourselves.
The strongest existing method in the full-parameter regime is CAT from the same paper. It reaches 29.2% accuracy under $\epsilon=8$ attacks, against 23.3% for standard adversarial training, while keeping clean accuracy "in the 80% range" on UCF101 with 3D ResNeXt-101. GAT reaches higher robust numbers: 81.9% under masked PGD. But it adds a full 3D GAN and its evaluation attack differs, so we treat it as a reference rather than a matched competitor. We run CAT ourselves on identical splits and clip lengths, with the same total budget of forward-backward passes and the same number of hyperparameter trials as our method. Beating it means one of two things: higher robust accuracy under unrestricted $\epsilon=8$ PGD at equal or better clean accuracy, or equal robustness at no more than a third of the passes.
The strongest competitor in the token regime is robustness tokens ported directly to video transformers and trained with full-band PGD (Robustness Tokens: Towards Adversarial Robustness of Transformers). On images it already preserves clean accuracy almost exactly, so it is the method most likely to make our accuracy argument unnecessary. It gets the same token count, the same losses, the same steps, and attack steps tuned with the same budget.
The ablation that isolates our key ingredient is full-band robustness tokens with the same per-video warm-start store. Without it, any cost saving could come from warm-starting alone rather than band-limiting. A second ablation adds a temporal-consistency feature loss to full-band tokens, to check whether the frequency constraint does anything beyond what a smoothness penalty does.
Why not just…?
Why not just shrink $\epsilon$ or use a curriculum, as CAT does? The evidence that weaker attackers help comes from exactly that paper, so this is the natural competitor. A smaller $\epsilon$, though, weakens the attacker uniformly along every direction, including the slow, static directions that Adversarial Framing for Image and Video Classification shows are dangerous on their own. Band-limiting keeps full strength along those directions and removes only the fake-motion ones. The $\omega$-curriculum against $\epsilon$-curriculum comparison at matched passes settles which is right. The order-sensitivity probe on SSv2 tells us why, depending on which method preserves the model's response to reversed clips.
Why not just port robustness tokens to video transformers unchanged? This is a real threat. With a frozen backbone and a loss that keeps clean features unchanged, clean degradation may already be small. That is why the first experiment checks for it before anything else. Even if clean accuracy holds, full-band tokens still pay a full PGD inner loop through a 16-frame transformer at every step. The warm-start ablation shows whether band-limiting buys cost savings that persistence alone does not.
Why not just cut cost with single-step attacks or attacks on a subset of frames? Single-step training is the obvious way to save passes. However, Analysis and Extensions of Adversarial Training for Video Classification found that video attacks are very sensitive to step size, which makes poorly tuned cheap attacks unreliable. Attacking only some frames creates a sparse perturbation with sharp temporal edges, which is the opposite of what we want. We include a matched-pass single-step full-band baseline. Our refresh is also one or two steps, so the comparison isolates the search space, not the step count.
Why not just keep the model fixed and add a temporal-consistency detector? AdvIT detects more than 95% of adversarial frames in real time, but it only raises an alarm and does not restore the prediction. It also depends on a flow network that Towards Understanding Adversarial Robustness of Optical Flow Networks shows can be broken. Low-band attacks are exactly the ones such detectors cannot see. We will report whether pairing our hardened model with a label-flip check in the style of Identifying and Resisting Adversarial Videos Using Temporal Consistency covers the high band. That pairing is a secondary result, not the main claim.
Proposed Experiments and Decisive Checks
Experiments use UCF101 and SSv2, with Kinetics-400 pretraining. The models are 3D ResNeXt-101 in the full-parameter regime and TimeSformer and MViT with robustness tokens in the cheap regime. Evaluation attacks are unrestricted PGD at $\epsilon \in \{4, 8\}$ with the reported step-size rule, static framing, universal 3D perturbations, flicker-only high-band PGD, and adaptive white-box attacks that differentiate through the tokens. The last addresses the concern that tokens act as a secret key. We evaluate robust accuracy on held-out clips, and for the adaptive attack we use more steps than training ever used.
Three outcomes decide the project. The first is transfer. If models trained with $\omega \le 2$ lose more than a few points of unrestricted-PGD accuracy relative to full-band training at matched compute, band-limited training does not cover the classifier threat model, and the claim fails outside a setting that also includes a detector. The second is mechanism. If full-band hardening does not reduce SSv2 order sensitivity more than band-limited hardening, the fake-motion explanation is wrong, even if the accuracy numbers favor us. The third is cost. If the warm-started full-band ablation matches our pass savings, the cost benefit belongs to persistence rather than to band-limiting, and we will say so.
A positive result on all three would give video practitioners a single interpretable knob, the temporal cutoff, along with a hardening recipe that is cheap enough for frozen video foundation models.
References
- Analysis and Extensions of Adversarial Training for Video Classification
- Adversarial Framing for Image and Video Classification
- Universal 3-Dimensional Perturbations for Black-Box Attacks on Video Recognition Systems
- Large-scale Robustness Analysis of Video Action Recognition Models
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- AdvIT: Adversarial Frames Identifier Based on Temporal Consistency in Videos
- Identifying and Resisting Adversarial Videos Using Temporal Consistency
- Towards Understanding Adversarial Robustness of Optical Flow Networks
- Structured Universal Adversarial Attacks on Object Detection for Video Sequences