Robust Video Domain Adaptation at the Cost of Ordinary Adaptation

Video models are routinely trained on one kind of footage and deployed on another. Typical cases are third-person web clips moving to drone footage, one kitchen moving to another, or game renderings moving to real cameras. The shift is in appearance, and it is also in how actions unfold over time. A separate line of work makes models resistant to adversarial perturbations, meaning tiny, deliberately chosen pixel changes that flip predictions. It does so through adversarial training, which repeatedly attacks the model during training and is therefore 7× to 30× more expensive than normal training. These two lines have barely met. What a deployed video model needs is robustness on the domain it is deployed in, and that domain has no labels.

This note proposes making a domain-adapted video model adversarially robust on its unlabeled target domain for roughly the price of the adaptation run alone. Video domain adaptation methods already run a backward pass through the whole network on every target clip, because they use target pseudo-labels and a domain classifier. The idea is to extend that backward pass one step further, to the input pixels, and recycle the resulting gradient as the adversarial perturbation for the next step. This is the "free" adversarial training trick, applied with pseudo-labels on unlabeled target video instead of labeled images. Because video perturbations are huge tensors, the perturbation is split into a part shared by all frames and a small per-frame part. Only the shared part is carried between steps.

Core Claim

We make two claims, and both can be falsified. First, adversarial robustness learned on the labeled source domain does not carry over intact across a temporal domain shift. Robust accuracy measured on target clips will fall well below robust accuracy on source clips, by more than clean accuracy falls. Robustness therefore has to be trained on the target. Second, target-side robustness can be obtained at a training cost within about 1.3× of a standard adaptation run. Its target robust accuracy should come close to an expensive multi-step attack baseline run on the same pipeline. We measure cost in forward and backward passes and in GPU-hours.

Grounding in Prior Work

The adaptation side supplies the base pipeline. MetaTrans separates motion from appearance architecturally. It uses a transformer stream with position information and a stream without it, and subtracts the second from the first. It trains with only two losses: source classification plus target pseudo-labels, and a domain-adversarial loss. It reports 95.4% average accuracy on UCF-HMDB. We build on it because its two-loss objective gives a clean single backward pass to piggyback on. Its "overhead" concern is the number of hyperparameter sweeps, not the cost of robustness training, and it never evaluates attacks. TranSVAE and TA3N motivate the static-versus-dynamic split and the observation that domain shift is uneven over time. TranSVAE disentangles static and dynamic latents with several losses. TA3N weights temporal alignment by the confidence of the domain classifier. Neither addresses input perturbations.

The robustness side supplies the cost-reduction method and its known failure modes. Adversarial training for free! updates the weights and the perturbation from the same backward pass. It replays each minibatch $m$ times and warm-starts the perturbation, reaching about 46% robust accuracy on CIFAR-10 at roughly natural-training cost. It notes that large $m$ hurts clean accuracy. Fast is better than free shows that a single attack step with random initialization reaches 45% on CIFAR-10 in 6 minutes. Without that randomization, training falls into catastrophic overfitting: the model resists the cheap one-step attack while collapsing under stronger multi-step attacks. Both methods require true labels and were validated only on images.

On video, FrameShield is the closest defense. It runs standard multi-step adversarial training over video chunks for anomaly detection and lifts UCSD Ped2 AUROC under attack from 2.4% to 81.3%. It works in a single domain, pays the full multi-step cost, and reports that frozen backbones cannot become robust. That last finding constrains our design: we fine-tune the backbone end to end. AstFocus shows that video attacks can concentrate on a few frames and patches. It reaches a 99% fooling rate on UCF-101/C3D with 2015 queries. We use it as an independent black-box check, not as a training method.

Research Program

The method runs inside MetaTrans training. Each minibatch holds source clips with labels and target clips with confident pseudo-labels. Pseudo-labels below a confidence threshold are masked. The perturbation on a clip with $T$ frames is parametrized as

$$\delta_t = s + r_t,\qquad \|s + r_t\|_\infty \le \epsilon,$$

where $s$ is one perturbation image shared by all frames and $r_t$ is a per-frame residual for frame $t$, held to a smaller budget. One forward and backward pass on the perturbed clips gives three things at once: gradients for the weights from the task and domain losses, the input gradient on $s$, and the input gradient on each $r_t$. Following the free-training recipe, the weights take a descent step, and both perturbation parts take a signed ascent step. The minibatch is replayed $m$ times. The total number of epochs is divided by $m$ so that the total pass count matches the plain adaptation run.

Across visits to the same clip, only $s$ is warm-started, and the $r_t$ are freshly randomly initialized, as in Fast FGSM. Storing a full warm-started perturbation for every frame of every clip would cost $T$ times as much memory. This makes the static/dynamic split an efficiency device as well as an analysis tool.

Three questions are studied alongside the main comparison. The first is which ascent objective the target perturbation should follow: the masked pseudo-label loss, the label-free domain-classifier loss, or both. The second is whether noisy pseudo-labels under domain shift speed up catastrophic overfitting. We track the gap between one-step and multi-step target robust accuracy every epoch, a question nobody has examined according to the gap analysis. The third is whether surviving attacks exploit the shared or the per-frame component. We answer it by attacking trained models with each component alone.

Baselines

The naive reference points are a source-only model and plain MetaTrans adaptation with no robustness training. They set clean accuracy and show how close target robust accuracy is to zero without defense. For clean accuracy, the reported reference is MetaTrans at 95.4% average on UCF-HMDB (MetaTrans), with TranSVAE at 93.37% on UCF-HMDB and 52.6% on Epic-Kitchens (TranSVAE). Our reruns under end-to-end pixel training may differ, and we will report our own numbers next to these.

The strongest competitor on robustness is joint multi-step adversarial training. Here a 7-step attack is run on both source clips and pseudo-labeled target clips inside the same MetaTrans pipeline, following the FrameShield-style chunk adversarial training (FrameShield). It directly attacks the deployment domain and does not skimp on attack strength. No reported result was found on Lacuna for adversarial robust accuracy on any video domain adaptation benchmark (UCF-HMDB, Epic-Kitchens, Kinetics→NEC-Drone), so this baseline must be run by us. FrameShield's 81.3% AUROC is on single-domain anomaly detection and is not comparable.

A cheaper and still strong competitor is two-pass Fast FGSM with random initialization on source and pseudo-labeled target clips (Fast is better than free). A sequential pipeline is also included: adversarial training on source, then standard adaptation. This tests whether robustness needs to be trained on the target at all.

Ablations isolate the key ingredients. The first uses full per-frame perturbations with no shared component, under the same pass budget. The second applies perturbations to source only, with no target perturbation. The third drives target perturbations with the domain loss only. A cheap parameter-efficient baseline adds Robustness Tokens on a frozen video transformer.

All methods use the same backbone, clip length, attack budget $\epsilon$, data, and a matched tuning budget of the same number of sweep runs per method. The main comparison plots target robust accuracy against GPU-hours. Beating the strongest baseline means one of two things. Either we match joint 7-step training's target robust accuracy within 2 points at at most a quarter of its GPU-hours, or we exceed two-pass Fast FGSM's target robust accuracy at a lower cost, without a drop in clean target accuracy of more than 2 points relative to plain MetaTrans.

Why not just…?

Why not just adversarially train on the labeled source and then adapt? This is the cheapest story, and it is a real threat if robustness transfers across the shift. Nothing in the file shows it does. VUDG reports that even strong video models drop 15.4 points out of domain, which suggests that decision boundaries move under temporal shift. The sequential baseline settles the question directly. If its target robust accuracy matches the joint methods, the first core claim fails and we will say so.

Why not just use two-pass Fast FGSM on pseudo-labeled target clips? This is the most serious alternative. It would plausibly reach similar robustness, and our advantage over it is mainly cost, roughly 1× versus 2× passes, plus perturbation memory. If replay-induced loss of data diversity, a documented weakness of free training, costs more clean accuracy on video than the saved pass is worth, Fast FGSM wins. The robustness-per-GPU-hour curve will show which one does.

Why not just randomize frames at inference time? Random resizing and padding (Mitigating Adversarial Effects Through Randomization) adds no training cost. The authors concede it is weaker against transferable attacks and can be bypassed by averaging attacks over the randomization. Per-frame jitter also disrupts the motion cues the adaptation relies on. We evaluate it under averaged-randomization attacks.

Why not freeze the backbone and train robustness tokens? This works on images, recovering DINOv2-G from 0.3% to 63.1% under white-box attack (Robustness Tokens). FrameShield, however, reports that frozen video backbones fail. We run it as a baseline, which also tests that contradiction directly in a cross-domain setting.

Decisive Checks

Experiments use UCF-HMDB, Epic-Kitchens, and Kinetics→NEC-Drone with end-to-end pixel inputs and 8–16-frame clips, kept short so that the 7-step baseline remains affordable. Robust accuracy is measured with frame-level $L_\infty$ attacks using 50-step attacks with restarts, applied to target clips.

Four checks decide the outcome. The robustness-transfer gap must appear in the sequential baseline. Our method must hit the cost and accuracy thresholds stated above. Robustness must not come from gradient masking: accuracy must keep falling smoothly as attack steps increase, and a query-based black-box attack in the style of AstFocus must not succeed more often than the white-box attack. Finally, the one-step versus multi-step gap must stay small throughout training. If it widens, catastrophic overfitting has occurred, and we will test whether a higher pseudo-label confidence threshold prevents it.