A single network that predicts several things from one image, such as semantic labels, depth and surface orientation, has two weaknesses. Its tasks compete for shared features. It can also be fooled by tiny, deliberately crafted pixel changes. Adversarial training is the standard defense: the model trains on the worst perturbations an attacker can find. Uncertainty-based loss weighting is the standard fix for task competition: each task's loss is divided by a learned noise level, which puts tasks measured in different units on a common scale. Combining the two naively fails in a specific way. The attacker used during training follows whichever task has the largest raw loss, so the smaller-scale tasks never see a real attack. The learned noise levels then read a task's large adversarial loss as "this task is noisy" and shrink its weight. Training ends up excusing exactly the task that is most vulnerable.
We propose fitting each task's scale only on clean losses. The same scale-normalized objective is then used twice: the training attacker maximizes it, and the model minimizes it. The task scales can be computed in closed form from a running average of clean losses, so the method adds one forward pass of bookkeeping and no extra backward passes. The test is worst-task robustness on NYUv2 and Cityscapes against balanced multi-task attacks that are known to break current defenses.
Core Claim
In multi-task adversarial training, the per-task weights do two different jobs. They set the attacker's objective in the inner maximization, and they set the learner's objective in the outer minimization. Prior work treats only one of these jobs.
Dynamic Gradient Balancing for Enhanced Adversarial Attacks on Multi-Task Models supplies the motivating evidence for the attacker side. It shows that attacking the plain sum of task losses on NYUv2 is dominated by segmentation and depth, which leaves surface normals almost untouched. It then shows that reweighting each task's attack gradient by inverse loss gives a much stronger attack, one that still degrades a model hardened with Friendly Adversarial Training by nearly 19% over baseline attacks. That paper is purely on the attacker's side, and so is Stealthy Multi-Task Adversarial Attacks, which tunes per-task weights to sabotage one task while keeping the others looking normal.
On the learner side, Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics supplies the method. It derives the loss $\sum_i \frac{1}{2\sigma_i^2}L_i + \log\sigma_i$ from a Gaussian likelihood with a per-task noise level $\sigma_i$. Its optimum sets $\sigma_i^2$ equal to the task loss. Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning shows that learning $\sigma_i$ by gradient descent lags the true loss by up to 100 epochs. It replaces the learned values with closed-form weights, a softmax over inverse losses. That softmax has the same inverse-loss form the attack paper uses.
Our observation is that the likelihood-normalized loss already is a scale-balanced attack objective. The attacker and the learner can therefore share one set of weights. The key design decision is where those weights are estimated from. If $\sigma_i$ is fit on adversarial losses, it inflates for the most attackable task and discounts its robustness gap. If it is fit on clean losses, it measures only the task's units and intrinsic noise, so a large adversarial gap remains fully penalized.
Concretely, let $L_i^{c}(\theta)$ and $L_i^{a}(\theta)$ be task $i$'s loss on clean and perturbed inputs. Let $s_i$ be an exponential moving average of $L_i^{c}$ that receives no gradient; this is the closed-form optimum of $\sigma_i^2$ under the Gaussian model. Training then solves
$$\min_\theta \; \mathbb{E}\Big[\sum_i \frac{L_i^{c}(x;\theta) + L_i^{a}(x+\delta^\star;\theta)}{s_i}\Big], \qquad \delta^\star = \arg\max_{\|\delta\|_\infty\le\epsilon} \sum_i \frac{L_i(x+\delta;\theta)}{s_i}.$$
Here $\epsilon$ is the perturbation budget. Because $s_i$ comes from clean data only, the effective weight on a task does not fall when the attack succeeds on it. Because the attacker uses the same normalization, every task is attacked in proportion to its own scale rather than its raw units.
Research Program
We would build the method on standard NYUv2 three-task (segmentation, depth, normals) and Cityscapes two-task (segmentation, depth/disparity) setups. Every arm uses a shared encoder with task heads, 10-step projected gradient descent attacks (iterated gradient steps clipped to the budget) at $\epsilon=8/255$ for training, and split batch normalization. Split batch normalization means separate normalization statistics for clean and perturbed inputs. Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off showed that mixing the two in shared statistics corrupts both. We hold this component fixed across all arms so it cannot explain the differences.
The analysis has three parts. First, we log, epoch by epoch, the share of the training attack's gradient norm attributable to each task. This tests whether the naive combination really leaves the smaller-scale task unattacked during training. Second, we log effective task weights under adversarially-fit versus clean-fit scales. This tests whether the former collapse the weight on the most vulnerable task. Third, we repeat the main comparison at two backbone sizes, a ResNet-50-scale and a ResNet-101-scale encoder. The analytical-weighting paper found that the advantage of adaptive weighting over equal weighting shrinks with capacity on clean data, while the attack paper found that more shared parameters make cross-task attacks transfer 23x to 31x more. Which effect dominates under adversarial training is open, and the outcome determines whether the method matters at scale.
The theory in Adversarially Robust Multi-task Representation Learning bounds how a jointly adversarially trained representation transfers robustness. It explicitly leaves open how to weight source tasks. Our method is one concrete answer for the joint-training phase. We also test the paper's premise with a transfer probe: we freeze each trained encoder, fit a new head for a held-out task (Cityscapes-trained encoder to a NYUv2 task, or the reverse), and measure that head's robust error.
Two related lines of work are deliberately outside our design. Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Training uses uncertainty to model the spread of adversarial features as a Gaussian in single-task classification, not to weight losses. On the robustness of adversarial training against uncertainty attacks shows adversarial training makes predictions less confident, treating uncertainty as an evaluation target. That finding constrains us: lower confidence after training means predictive entropy is not a usable weighting signal. This is one reason we derive scales from clean task losses rather than from output entropy.
Baselines
The naive reference point is equal-weight multi-task adversarial training. It sums raw clean and adversarial losses across tasks, and its training attack maximizes the raw total loss. This is what a practitioner would write first, and it is the setting in which the attack paper shows normals going undefended.
The obvious combination of the two directions is Kendall-Gal uncertainty weighting with learned $\sigma_i$ applied to the adversarial losses, keeping the raw-total-loss training attack. It tests whether the uncertainty paper's method alone suffices. Swapping in the closed-form weights from the analytical-weighting paper, fit on adversarial losses, gives a third baseline that removes update lag but keeps the "excuse the vulnerable task" behavior.
The strongest competitor is Dynamic Gradient Balancing used as the training attacker, with equal outer weights. This is adversarial training against the strongest known multi-task attack. It directly targets the attacker-side half of our claim at comparable cost.
For gradient-level conflict resolution we include Conflict-Aware Adversarial Training. It projects the adversarial gradient into a cone around the clean gradient, and we extend it to apply per task. We report wall-clock cost alongside accuracy, because it needs separate backward passes.
As an oracle we run a small static grid over per-task weights. Its size is matched to the total compute of our method's runs plus tuning.
All arms share data splits, architecture, attack budget, number of epochs, split normalization, and an equal hyperparameter-search budget measured in GPU-hours. All are evaluated with the same attack suite.
There are no reported numbers for multi-task adversarial training defenses on NYUv2 or Cityscapes in the material we have. The reference points are therefore attack-side figures on undefended models. On NYUv2, Multi-Task Adversarial Attack reduced segmentation accuracy from 53.50% to 6.27% and nearly tripled depth error. On Cityscapes, the stealthy-attack paper reports a PGD $\ell_\infty$ attack at $\epsilon=8/255$ dropping mIoU from 54.20 to 11.31 while disparity error rose from 81.51 to 164.19. For the strongest defense comparison, the only reported number is Friendly Adversarial Training losing nearly 19% more under Dynamic Gradient Balancing than under baseline attacks. We will reproduce that defense as an additional baseline.
Beating the strongest competitor means matching or exceeding Dynamic-Gradient-Balancing adversarial training on worst-task robust error under all evaluated attacks. It also means improving average clean performance relative to single-task models, measured by the standard relative multi-task metric, by a margin larger than the seed-to-seed spread over three seeds, at no more than 1.1x its wall-clock cost.
The ablation that isolates the key ingredient keeps everything in our method fixed but fits $s_i$ on adversarial instead of clean losses. A second ablation keeps clean-fit scales in the outer loss but uses the raw-total-loss attacker. Together these separate the "where scales come from" effect from the "shared attacker objective" effect.
Why not just…?
Why not just use equal weights and a bigger backbone? On clean data, the analytical-weighting paper found that the gap between equal and adaptive weighting narrows at ResNet-101 scale, so this is a real threat. The attack paper's finding that more shared capacity makes cross-task attacks transfer more points the other way. Our two-backbone comparison settles it. If equal weighting matches our method on worst-task robust error at the larger backbone, we will report that the method matters only at small capacity.
Why not just grid-search static weights? For two or three tasks a grid is feasible, so we include it as a compute-matched oracle. The argument against it is not tractability at this scale. It is that a static weight cannot follow the shifting balance between clean and adversarial gradient conflict that the conflict-aware paper documents as the budget grows. If the grid matches us at matched compute, our contribution narrows to removing the need to tune. We will say so.
Why not just train against Dynamic Gradient Balancing? This is the most serious alternative, because it fixes the attacker-side problem directly. It does not fix the learner side. Its outer loss is still raw and unbalanced, so the normals head may be attacked well yet still receive too little weight. The ablation pairing clean-fit outer scales with the raw-total-loss attacker, compared against this baseline, tells us which half carries the gain. Its attack also requires solving a per-step integer program, which we will time. If it wins outright, the right conclusion is that only the attacker-side weighting matters.
Why not just use gradient surgery? Conflict-aware projection addresses gradient direction, not scale. It also needs one backward pass per objective, which with three tasks plus clean and adversarial terms multiplies cost. The figures in the invariance-regularization paper show how costly overhead already is in adversarial training: TRADES runs 1.51x slower than standard adversarial training. We report both accuracy and time, and count our method as better only if it matches the projection method's worst-task robustness at clearly lower cost.
Proposed Experiments
We evaluate every model on NYUv2 and Cityscapes at $\epsilon\in\{4,8,16\}/255$ against four attacks. The first is the raw-total-loss PGD attack. The second is Dynamic Gradient Balancing. The third is a per-task targeted PGD attack against each task in turn, which gives the worst-task metric. The fourth is the stealthy attack, scored on whether non-target metrics stay inside a detection band while the target task fails. We use layer6ai-labs/ssl-robustness as a cross-check harness for NYU depth robustness. Metrics are mIoU, depth absolute relative error, mean angular error for normals, the relative multi-task gain over single-task models, worst-task robust degradation, and wall-clock time per epoch.
Three checks would be decisive. First, the gradient-share logs must show the naive training attack leaving normals mostly unattacked, and ours must not. If both attack all tasks evenly, the attacker-side argument fails. Second, the clean-fit versus adversarial-fit ablation must show the adversarial-fit variant losing robustness specifically on the most vulnerable task. If the two are indistinguishable, the learner-side argument fails. Third, the head-to-head against Dynamic-Gradient-Balancing adversarial training at both backbone sizes decides whether the full method is worth adopting or whether the field should simply train against the stronger attack.
References
- Dynamic Gradient Balancing for Enhanced Adversarial Attacks on Multi-Task Models
- Stealthy Multi-Task Adversarial Attacks
- Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
- Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
- Adversarially Robust Multi-task Representation Learning
- Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Training
- On the robustness of adversarial training against uncertainty attacks
- Conflict-Aware Adversarial Training
- Multi-Task Adversarial Attack