Adversarial training makes a network resistant to small, deliberately chosen input changes. It does this by training on perturbed inputs, and finding each perturbation takes several extra gradient passes per image, which multiplies training cost roughly tenfold. For a generalist, meaning one network with a shared backbone and several task heads (segmentation, depth, surface normals), this cost is usually treated as an extra burden on a model that already trails its single-task specialist counterparts. This note argues the opposite accounting. The expensive part, the input-space gradient through the backbone, is paid per image, not per task. A shared-backbone generalist could therefore pay the adversarial cost once for all its tasks, while $T$ specialists pay it $T$ times. Whether that saving is real depends on one question: can a single cheap perturbation per image be strong enough for every task at once?
Prior work says the obstacle is sharp. In a multi-task model, adversarial vulnerability is set by the weakest task: one fragile head compromises the shared encoder for everyone (Adversarial Robustness in Multi-Task Learning: Promises and Illusions). Deeper sharing also makes attacks transfer across tasks 23x to 31x more readily (Dynamic Gradient Balancing for Enhanced Adversarial Attacks on Multi-Task Models). We propose to turn both findings into a training rule. At each step, the generalist uses cheap head-level gradients to find the task nearest to failure on that image. It then takes one analytically sized input step against that task, and relies on the strong cross-task transfer to cover the rest. We also measure the robust generalist–specialist gap at matched compute, which to our knowledge nobody has reported.
Core Claim
A shared-backbone multi-task network can reach adversarial robustness comparable to multi-step training on every task. It needs only about one extra backbone backward pass per image, provided three things hold. The perturbation must target the currently weakest task. It must be sized by a first-order estimate of that task's distance to failure. And it must extend to regression tasks such as depth and normals, which have no classification margin. A further prediction follows: the more layers the tasks share, the better one perturbation covers all of them. The sharing that Dynamic Gradient Balancing identified as a vulnerability should become a cost advantage during training.
Method
The step-size idea comes from Single-Step Adversarial Training for Semantic Segmentation. That paper's Fast Newton Method sizes a single adversarial step from a margin-based "gain" and a first-order Taylor estimate of how far the input is from the decision boundary. With one extra gradient pass, it avoids catastrophic overfitting, the failure where single-step training learns to beat only its own weak attack. That work is single-task segmentation only. We extend it in two ways.
The first extension gives every task a differentiable gain $g_i(x)$ that is positive while task $i$ is still acceptable on image $x$ and crosses zero at failure. For segmentation this is the mean pixel classification margin, as in the Fast Newton Method. For depth and normals we define $g_i = \tau_i - e_i(x)$. Here $e_i$ is the mean per-pixel error, and $\tau_i$ is a tolerance set to the clean validation error inflated by a fixed factor. This choice of $\tau_i$ puts the notion of "failure" on a comparable relative scale across tasks.
The second extension is a cheap triage step that picks the weakest task. Let $h = \phi(x)$ be the shared backbone features. We compute $u_i = \nabla_h g_i$ for all tasks, which needs only backward passes through the heads. We then take the task with the smallest feature-space distance to failure, $k = \arg\min_i g_i / \|u_i\|$. A single input-space backward pass gives $v = \nabla_x g_k$. The perturbation is
$$\delta = -\alpha\,\mathrm{sign}(v), \qquad \alpha = \min\!\left(\epsilon,\; \frac{g_k(x)}{\|v\|_1}\right),$$
where $\epsilon$ is the $\ell_\infty$ attack budget. The step size comes from the first-order approximation $g_k(x+\delta) \approx g_k(x) - \alpha\|v\|_1$, which predicts the smallest sign step that drives the weakest task to failure. The model is then trained on the summed task loss at $x+\delta$. Relative to standard training, the cost is one forward pass, the head-only backward passes, and one extra full backward pass. Multi-step training with $K$ steps costs about $K+1$ times standard training.
Research Program
The work has three parts. First, a measurement study on NYUv2 (segmentation, depth, normals) and Tiny-Taskonomy (five dense tasks). We train shared-encoder models at three or four sharing depths, following the architecture family in Dynamic Gradient Balancing, alongside matched specialists. We report clean and robust performance per task and total training GPU-hours. This gives the robust generalist–specialist gap at matched compute. We expect the clean gap to resemble the one UniT reports (39.0 versus 43.3 mAP against DETR on COCO, from UniT). It is an open question whether the gap widens or narrows under attack.
Second, the method above, with ablations on each ingredient. Third, a transfer analysis. For every trained model, we perturb against each task alone and record the damage to every other task. This tests whether single-target training covers non-targeted tasks, and how that coverage scales with sharing depth.
All robustness claims will be evaluated with adaptive attacks designed for multi-task models. Promises and Illusions showed that naive PGD and task-weighting create gradient masking, meaning the model hides its gradients so attacks appear to fail without it being truly robust. Its weighted-gradient attack (WGD) was twice as effective as PGD on normal estimation. Dynamic Gradient Balancing's inverse-loss-weighted attack (DGBA) beat total-loss attacks by up to 80.41% on clean models and nearly 19% on adversarially trained ones. Our primary threat model is therefore multi-step DGBA and WGD with many restarts, in the spirit of the standardized evaluation argued for by RobustBench. That work found published defenses overestimated robustness by more than 10%.
Baselines
The naive reference points are the standard-trained generalist and, for cost, plain single-step FGSM on the summed task loss, with and without random initialization. These establish how much robustness is free and whether catastrophic overfitting appears in the multi-task setting. Single-Step Adversarial Training for Semantic Segmentation observed it in single-task segmentation.
The strongest existing method is multi-step adversarial training of the generalist, with the perturbation computed by DGBA on the balanced multi-task loss. We will also run Friendly Adversarial Training, the defense evaluated by Dynamic Gradient Balancing. This is the most robust recipe the file describes for multi-task dense models, and it is the compute ceiling we aim to approach. No reported result was found on Lacuna for absolute robust mIoU, depth error, or normal error of multi-step-trained NYUv2 or Tiny-Taskonomy models. The only reported figure is the relative one: DGBA is nearly 19% more damaging than total-loss attacks against FAT-defended models. We will therefore reproduce this baseline ourselves.
A third baseline is the Fast Newton Method applied off the shelf to the summed loss, which requires a margin for regression tasks. We will provide it the same gains we define. This is the most dangerous competitor, because it isolates whether weakest-task triage adds anything. Fourth, we run a set of specialists, each trained with multi-step and with single-step Fast Newton adversarial training. Fifth, we freeze a pretrained ViT backbone and train Robustness Tokens, a few learned vectors prepended to the image patches, following Robustness Tokens. We test both one shared token set and one set per task. That paper reports ADE20K mIoU under attack recovering from 4.7 to 23.3 on DINOv2, in single-task use.
Fairness rules apply across all baselines. Every method uses the same data, backbone, attack budget $\epsilon$, number of epochs, and a hyperparameter search of equal size. Cost is reported as backbone backward passes and wall-clock GPU-hours. We claim success if our method matches multi-step DGBA training in worst-task relative robust drop, within the seed-to-seed spread, at under a third of its compute. It must also beat summed-loss Fast Newton training on worst-task robustness at equal compute. Two ablations isolate the key ingredient: replacing triage with random task selection, and replacing triage with a DGBA-weighted gradient blend in a single step.
Why not just…?
Why not just use FGSM on the summed loss? In single-task segmentation it overfits catastrophically: the model learns to defeat the weak training step while staying open to multi-step attacks (Single-Step Adversarial Training for Semantic Segmentation). The summed loss adds another problem, because it is dominated by whichever task has the largest loss scale (Dynamic Gradient Balancing). Our naive baseline tests this directly, and robust accuracy curves over training will show whether overfitting occurs.
Why not just run the Fast Newton Method on the summed or reweighted loss? This is a real threat. If it matches our method under DGBA and WGD, triage is unnecessary, and the contribution shrinks to the matched-compute gap study plus the regression gains. The weakest-link evidence from Promises and Illusions predicts that a summed-loss step will leave the most fragile task undertrained. The head-to-head comparison at equal compute settles this. Loss reweighting alone is not a credible defense, because the same paper shows it produces gradient masking.
Why not just freeze the backbone and train Robustness Tokens? This is far cheaper, converging in 200 steps per Robustness Tokens. However, that work relies on the tokens staying secret and was tested only one task at a time. We will evaluate the tokens under full white-box access. If they hold up across all tasks at once, that is a cheaper answer, and we will report it as such.
Why not just deploy specialists? They avoid cross-task contamination but cost more parameters: 1.6B versus 201M in UniT. They also pay the adversarial cost $T$ times. The matched-compute comparison measures exactly whether their robustness advantage survives that cost.
Decisive Checks
The project succeeds or fails on four measurements. The first is worst-task robust drop under multi-step DGBA and WGD, as a function of training compute, for all methods on NYUv2 and Tiny-Taskonomy. The second is whether the robustness of tasks our method never targets improves as sharing depth increases. This is the transfer prediction. If it is flat or negative, the core claim fails. The third is whether robust accuracy under strong attacks keeps improving over training, with no late collapse that would indicate catastrophic overfitting. The fourth is a Cityscapes reproduction of the Fast Newton Method, confirming our implementation matches the reported single-task behavior before we extend it to multiple tasks.
References
- Adversarial Robustness in Multi-Task Learning: Promises and Illusions
- Dynamic Gradient Balancing for Enhanced Adversarial Attacks on Multi-Task Models
- Single-step Adversarial Training for Semantic Segmentation
- UniT: Multimodal Multitask Learning with a Unified Transformer
- RobustBench: a standardized adversarial robustness benchmark
- Robustness Tokens: Towards Adversarial Robustness of Transformers