A multi-task network uses one shared backbone to serve several prediction heads, for example segmentation, depth and surface normals from a single image. Making such a network robust to adversarial perturbations (small, deliberately crafted input changes that break the model) is expensive. Adversarial training runs an inner attack loop at every step, multiplying training cost by roughly 10 to 40. With many tasks it also raises a question nobody has answered well: which task should the attack target? The cheap default builds one attack on the summed task losses. Attack studies show that this sum is dominated by whichever task has the largest gradients, so the other tasks are trained against weak adversaries. Attacking every task separately fixes this but multiplies the attack cost by the number of tasks.
We propose to treat the adversary's budget as a resource shared across tasks and to allocate it, one task per batch, to the task whose attacks will do the most good. Two findings from the literature make this possible. The first is that a multi-task network is only as robust as its weakest task. The second is that, through the shared backbone, perturbations aimed at one task transfer heavily to the others. The second finding is normally read as a vulnerability. Here it is a saving: an attack aimed at the right task also trains the other tasks to resist attack. The core claim is that a scheduler combining two signals, how vulnerable each task currently is and how well attacks transfer between tasks, can match the worst-task robustness of per-task attacks at the cost of a single attack per batch. It should also avoid the false robustness that loss reweighting produces.
Where This Sits
Three results motivate the proposal. Adversarial Robustness in Multi-Task Learning: Promises and Illusions shows on NYUv2 that adding one vulnerable task, such as segmentation, degrades robustness of the whole shared encoder (the "one bad apple" effect). It also shows that robustness apparently gained by task reweighting is often gradient masking: the model hides its gradients, so standard attacks fail even though the model is not actually robust. An adaptive attack that reweights task gradients (Weighted Gradient Descent, WGD) was twice as effective as standard projected gradient descent (PGD, the iterative attack used inside most adversarial training) on normal estimation. Dynamic Gradient Balancing for Enhanced Adversarial Attacks on Multi-Task Models shows that summed-loss attacks fail because of gradient dominance, and that its balanced attack (DGBA) still beats baselines by nearly 19% against Friendly Adversarial Training. It also reports that more shared layers raise cross-task attack transferability by 23x to 31x on Tiny-Taskonomy. Both papers study attacks. Neither builds a training procedure, and neither uses transferability as a signal for allocating compute.
Two results shape the method. GAT: Guided Adversarial Training with Pareto-optimal Auxiliary Tasks shows that auxiliary tasks can regularize robust training. It resolves task trade-offs with a multi-objective gradient solver whose overhead the authors themselves flag, and it depends on hand-picked auxiliary tasks. Our scheduler replaces the solver with a cheap per-batch choice, and its transfer estimate is a data-driven signal about which auxiliary tasks help. Conflict-Aware Adversarial Training treats clean and adversarial objectives as two conflicting tasks and projects the adversarial gradient onto a cone around the clean one. It is evaluated only on single-task classification. We borrow its projection as one component to be tested by ablation, not as the contribution.
Two further results constrain the design. The masking results in the illusions paper rule out any defense evaluated only against summed-loss PGD, so every claim here is tested against WGD and DGBA. Gradient surgery in the style of Gradient Surgery for Multi-Task Learning (PCGrad) needs a separate backward pass per objective and pairwise projections. Applied to a clean and an adversarial gradient for each of $T$ tasks, it would stack a per-task cost on top of the attack loop. We deliberately avoid that design. The theory in Adversarially Robust Multi-task Representation Learning says pooling source tasks shields a data-poor target task. It leaves open how to choose and weight source tasks, and our scheduler is one practical answer to that question.
Core Mechanism
Let $\ell_j(x)$ be the loss of task $j$ on input $x$. Let $\delta_a$ be a PGD perturbation that maximizes $\ell_a$ alone. At each batch, the model trains on all task losses, clean and perturbed, at $x+\delta_a$ for one selected task $a$. This costs one attack plus one forward pass through all heads, the same as summed-loss adversarial training.
Every $N$ steps, on a small probe batch, we estimate a transfer matrix $M_{aj}$. It is the rise in $\ell_j$ caused by $\delta_a$ divided by the rise caused by $\delta_j$, so it measures how much an attack on $a$ hurts task $j$ compared with a dedicated attack on $j$. We also track each task's vulnerability gap $v_j$, the relative drop from clean to robust performance on held-out data under the balanced attack. The task to attack is sampled with probability proportional to
$$ s_a = \sum_{j=1}^{T} v_j \, M_{aj}, $$
a softmax over these scores with a temperature. The score favors attacks that are both aimed at the weak tasks and transfer to them. Computing $M$ needs $T$ extra short attacks on a probe batch every $N$ steps. With $N$ around 100 and probe batches of about one-sixteenth size, this amortizes to under 5% overhead. The illusions paper reports a 0.94 Pearson correlation between ResNet-18 and ResNet-50 robustness, so we also test estimating $M$ on a small surrogate network, which would make the estimate nearly free.
The optional conflict component compares the shared-encoder gradient of the adversarial losses with that of the clean losses, both aggregated over tasks. When their cosine is negative, it projects the first onto a cone around the second, following CA-AT. This adds one extra backward pass through the encoder, not $2T$.
Research Program
The work has three parts. The first is diagnostic. On NYUv2, we train multi-task networks with summed-loss PGD and measure per-task robustness under summed PGD, WGD and DGBA. The expectation is that dominated tasks look robust under summed PGD and collapse under the adaptive attacks. This would directly show negative transfer inside the adversary itself, and it is the premise the method rests on.
The second part builds the scheduler and measures how $M$ evolves during training. A key question is whether transfer stays high as the backbone becomes robust or decays. If it decays, the scheduler must drift toward per-task attacks, and the savings shrink. We also record negative transfer in the ordinary sense: per-task clean and robust performance relative to single-task robust models trained with matched compute.
The third part tests the method in the auxiliary-task regime of GAT on CIFAR-10. There, the transfer matrix should reveal which auxiliary tasks carry robustness to the main task, and the scheduler should down-weight useless auxiliaries without manual selection.
Baselines
The naive reference point is multi-task adversarial training with one PGD attack on the summed task losses. It has exactly our per-step cost and is what practitioners do by default. On NYUv2, the relevant reported numbers concern undefended models. Multi-Task Adversarial Attack reports segmentation accuracy falling from 53.50% to 6.27% under attack. We found no reported result on Lacuna for adversarially trained multi-task networks on NYUv2 or Tiny-Taskonomy, so we train all of these baselines ourselves. We use the same backbone, the same number of epochs, and the same hyperparameter search budget for every method.
The strongest compute-matched competitor is balanced-adversary training. It runs one PGD attack per batch on inverse-loss-weighted task losses, following DGBA's weighting, in two variants: with and without DGBA's integer program. This is the real threat to our method, because it also targets dominance at single-attack cost. It differs from ours in balancing all tasks instead of concentrating on the weakest one.
The strongest method regardless of cost is per-task PGD. It generates a separate attack for each task and costs roughly $T$ times our attack budget. It sets the robustness ceiling we aim to match. We also include GAT on CIFAR-10 with auxiliary tasks. GAT reports 56.21% robust accuracy for ResNet-50, and we rerun it under our evaluation.
Ablations isolate the key ingredient. We replace $s_a$ with uniform task sampling, which tests whether scheduling matters at all. We use vulnerability $v_j$ alone without $M$, which tests whether transfer information matters. We remove the cone projection. We estimate $M$ on a surrogate network instead of the model itself.
Beating the baselines means two things. First, worst-task robust performance under the strongest of WGD and DGBA must be significantly higher than both summed-loss and balanced-adversary training, measured over at least three seeds, at matched wall-clock time. Second, it must come within a small margin of per-task PGD while using about $1/T$ of its attack compute. Mean-over-task gains without a worst-task gain do not count.
Why not just…?
Why not just train separate robust single-task models? This avoids cross-task contamination entirely. It costs $T$ backbones, however, and in low-data settings it forfeits the robustness that auxiliary tasks provide. GAT reports CheXpert robust AUC rising from 50% under standard adversarial training to 83% with auxiliary tasks. We include single-task robust models at matched total compute on NYUv2. If they win on worst-task robustness, that is a meaningful negative result for multi-task robust training, and we will report it as such.
Why not just use the balanced (inverse-loss) adversary? This is the alternative most likely to match us, and we treat it as the main competitor rather than a strawman. Balancing removes dominance but spreads the attack budget evenly. The one-bad-apple effect suggests that robustness is limited by the weakest task, so concentrating on it should help. The comparison is settled empirically on worst-task robustness. If balanced adversaries match the scheduler, the transfer matrix is unnecessary and the contribution reduces to the diagnostic finding that summed-loss training is illusory.
Why not just apply CA-AT's projection to the summed multi-task loss? Projection fixes the conflict between clean and adversarial updates. It does nothing about which task the attack was aimed at, so dominated tasks still see weak adversaries. The ablation that keeps the projection but uses summed-loss attacks tests this directly.
Why not just drop the vulnerable task? Dropping the bad apple sacrifices a required output. It also helps only if the vulnerability of other tasks comes from that task's gradients rather than from shared-backbone sensitivity that any task can expose. The diagnostic study retrains without segmentation to check which explanation holds.
Proposed Experiments
The main testbeds are NYUv2 (segmentation, depth, normals) and Tiny-Taskonomy (five tasks), which the file's attack papers already use. Tiny-Taskonomy stresses how the scheduler scales with task count. We add CIFAR-10 with rotation and jigsaw auxiliary tasks in the GAT setting, including the 10%-data regime.
Metrics are reported per task for clean and robust performance: mIoU for segmentation, error for depth, and angular error for normals. The headline number is worst-task relative robust degradation under the strongest of PGD, WGD and DGBA, computed with attack parameters tuned against each defense. Compute is reported as wall-clock time and as counts of forward and backward passes.
Three checks are decisive. First, summed-loss adversarial training must show a large gap between its apparent robustness under summed PGD and its robustness under WGD or DGBA on dominated tasks. Without that gap, the problem we target is not real. Second, the scheduler must close most of the gap to per-task PGD at single-attack cost. It must also beat both balanced-adversary variants on worst-task robustness. Third, the transfer matrix must predict gains. Training with a forced single attack task $a$ should improve task $j$'s robustness roughly in proportion to $M_{aj}$, a correlation we measure across all task pairs on NYUv2. If that correlation is weak, the mechanism behind any gain is something other than transfer, and we would say so.
References
- Adversarial Robustness in Multi-Task Learning: Promises and Illusions
- Dynamic Gradient Balancing for Enhanced Adversarial Attacks on Multi-Task Models
- GAT: Guided Adversarial Training with Pareto-optimal Auxiliary Tasks
- Conflict-Aware Adversarial Training
- Gradient Surgery for Multi-Task Learning
- Adversarially Robust Multi-task Representation Learning
- Multi-Task Adversarial Attack