When one network learns several dense prediction tasks at once (labeling every pixel with a class, estimating its depth, predicting its surface orientation), someone has to decide how much each task's loss counts. Today that choice is made on data from the same distribution the model was trained on, either by grid search or by an automatic balancer. This proposal tests a simple suspicion and builds on it. Tasks degrade at different rates when the input distribution shifts, for example from a driving simulator to real streets. If so, the weights that look best on in-distribution validation data are the wrong weights for deployment. The plan is to measure this directly. If it holds, we replace the in-domain balancing signal with one computed on a source domain the model is not currently training on, rotated across sources during training, so the weights are tuned for transfer rather than for fit.
The opportunity exists because two literatures have not met. Multi-task loss balancing is evaluated almost entirely in-domain on NYUv2, Cityscapes and CelebA. Domain generalization, meaning training on several source domains and testing on an unseen one, is evaluated almost entirely on single tasks. A balancer that only ever sees source-distribution losses cannot know that, say, segmentation collapses on real images while depth transfers well. A held-out source domain gives a cheap proxy for that knowledge without touching target data.
Core Claim
In multi-task dense prediction under domain shift, loss weights are a domain-generalization hyperparameter, not just an optimization one. Concretely, we claim three things. First, the ranking of static weight settings by in-domain validation score correlates poorly with their ranking by unseen-domain score. Second, a first-order bilevel balancer whose upper level scores task losses on a held-out source domain recovers most of the gap. Third, it does so at a fraction of the cost of the honest alternative, which is grid search with leave-one-domain-out selection. The first claim can fail on its own, and if it does, the method is unnecessary. That would itself be a useful negative result.
Relation to Prior Work
The method borrows its machinery from Scalable Bilevel Loss Balancing for Multi-Task Learning (LDC-MTL). That paper updates task weights in an outer loop to shrink loss gaps between tasks, at constant cost in the number of tasks. It evaluates only in-domain, and its outer objective is measured on the same distribution as training. We keep its single-loop, first-order structure and change what the outer loop looks at. The idea of meta-learning a loss on a held-out domain comes from Loss Function Learning for Domain Generalization by Implicit Gradient. That work learns the shape of a single classification loss on RotatedMNIST. We instead learn the relative weights of heterogeneous losses (cross-entropy plus continuous regressions), which its scalar polynomial cannot express. The rotation of held-out sources follows the episodic logic of Episodic Training for Domain Generalization, which pairs modules with "mismatched" source domains for image classification. That method itself leaves a balancing weight to manual tuning, which is exactly the hyperparameter we target.
Two papers motivate the gap. Multi-Task Learning for Dense Prediction Tasks: A Survey reports that carefully tuned grid search often beats GradNorm, MGDA and uncertainty weighting. That finding constrains us: any automatic balancer must beat a well-tuned static baseline, not a naive one. Domain Generalization for Semantic Segmentation: A Survey documents that dense domain generalization is studied single-task, and warns that foundation-model backbones blur what "unseen" means. Three further papers constrain the design. Semantic-Aware Domain Generalized Segmentation shows that global style normalization destroys class cues in dense outputs. We therefore do not make feature normalization our lever, and we treat their method as an orthogonal add-on. Domain Generalization via Gradient Surgery (Agr-Sum) and Gradient Matching for Domain Generalization (Fish) operate on per-domain gradients of a single task. Gradient Surgery for Multi-Task Learning (PCGrad) and RotoGrad operate on per-task gradients within one domain. None of these considers the task-by-domain grid. Finally, Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models (FAMDA) is the closest multi-task, cross-domain dense work. It uses unlabeled target images, so it addresses adaptation rather than generalization, and it supplies our benchmark pairing.
Research Program
The first phase is diagnostic and decides whether the rest is worth doing. We train a shared-encoder, two- or three-head model on synthetic source domains over a grid of static task weights. For each setting we record both in-domain validation scores and scores on unseen real target domains. We then compute the rank correlation between the two orderings, along with the per-task degradation ratio, which is the target score divided by the in-domain score. In parallel we log the full task-by-domain gradient cosine matrix during training. This tells us whether conflicts are mainly across tasks, across domains, or entangled.
The second phase builds the balancer. Let $S$ be the set of source domains, $w$ the vector of task weights, $\theta$ the network parameters, and $L_{t,d}$ the loss of task $t$ on domain $d$. At each episode one source $h \in S$ is held out. The inner problem trains on the rest, and the outer problem scores the result on $h$:
$$\theta^*(w) = \arg\min_\theta \sum_{d \in S \setminus h} \sum_t w_t L_{t,d}(\theta), \qquad \min_w \sum_t \frac{L_{t,h}(\theta^*(w))}{\bar L_{t,h}}.$$
Here $\bar L_{t,h}$ is a running reference loss for task $t$ on $h$, which puts continuous and discrete losses on a common scale. Following LDC-MTL, $\theta^*$ is approximated by the current iterate, and $w$ takes one first-order step per iteration using a one-step lookahead gradient. The overhead is one extra forward and backward pass on a held-out batch, independent of the number of tasks. Source domains that label only some tasks contribute only those terms. We also test a variant that sends the outer gradient to per-task-per-domain weights $w_{t,d}$, which directly asks whether the two-dimensional conflict structure matters.
Baselines
The naive reference is pooled training on all sources with equal task weights and no balancing. Every method must clear it. A second reference is single-task models trained per task on the same pooled sources. This defines the multi-task gain Δm and tells us whether sharing helps at all under shift.
The strongest competitor on this question is static grid search with leave-one-source-domain-out selection. We use the same weight grid as Phase 1 and choose weights by average held-out-source score, never target score. The dense-prediction survey shows tuned grids beat automatic balancers in-domain, and this variant uses exactly our held-out signal, just non-adaptively. It gets the same backbone, schedule and data. Its tuning budget is reported in GPU-hours, and we also run a budget-matched version whose grid costs the same as one run of our method plus its own small search. Beating it means better target Δm at equal compute, or equal Δm at a several-fold lower cost. Merely matching it at equal compute would not count.
The strongest published balancer is LDC-MTL, run unchanged with its outer loop on in-domain validation batches. This is also the key ablation, because it differs from our method only in which data the outer loop sees. For reference, LDC-MTL reports Δm = −1.31% on CelebA against 0.37% for FairGrad, in-domain. No reported result was found on Lacuna for LDC-MTL, PCGrad or any multi-task balancer on a synthetic-to-real multi-task benchmark. We also run PCGrad, and a two-stage surgery that applies Agr-Sum across domains and then PCGrad across tasks, as the gradient-manipulation alternatives.
For segmentation quality under shift, the anchor is Semantic-Aware Normalization and Whitening, which reports 39.75% mIoU on GTA5 → Cityscapes with ResNet-50 against a 29.32% baseline (single-task). We add it to both our method and the grid baseline, to show the gains stack rather than substitute. No reported result was found on Lacuna for multi-task domain generalization on SYNTHIA/VK2 → Cityscapes, so all multi-task numbers on that pairing will be produced in this work.
Why not just…?
Why not just grid-search static weights with leave-one-domain-out selection? This is the real threat, and the proposal is built so it can win. If Phase 1 shows that held-out-source rankings predict target rankings well, and the budget-matched grid equals our method, then the adaptive balancer is not needed. The contribution would shrink to the finding that weights should be selected on held-out domains, and we would report it that way. The method's case rests on two things. One is cost: the grid grows exponentially with task count, while our balancer adds one pass. The other is that the best weights may change during training, which no static setting can follow. The per-epoch trajectory of learned weights will show whether this happens.
Why not just pool everything and use equal weights? In single-task domain generalization, pooled training already loses to gradient-aware training. Mansilla et al. report 63.89% versus 65.80% on PACS, and Fish reports 70.3% versus 74.7% on Camelyon17. Whether the same holds for multi-task dense outputs is unknown, and the pooled baseline measures it directly.
Why not just apply PCGrad or Agr-Sum? Mansilla et al. found PCGrad less effective than agreement filtering for domain shift. Agr-Sum, in turn, zeros coordinates, which may erase the weak gradients of secondary dense heads. Both also need one backward pass per objective, and the task-by-domain grid multiplies that number. The two-stage surgery baseline tests whether careful gradient editing gets what we get more cheaply.
Why not just use a foundation backbone and ignore balancing? Foundation backbones reach above 71% mIoU on Cityscapes (MFuser, per the segmentation survey), but they have likely seen real street imagery, so the "unseen" test is compromised. We run both an ImageNet-initialized ResNet-50 and a MiT-B2 encoder. We report whether the in-domain versus out-of-domain weight mismatch shrinks with stronger pretraining. If it does, that is evidence the problem fades as pretraining scales.
Proposed Experiments
The primary benchmark uses SYNTHIA and Virtual KITTI 2 as sources with segmentation and depth labels, following the pairing in FAMDA, with Cityscapes as the unseen target. GTA5 is added as a segmentation-only third source to test partially labeled domains. BDDS and Mapillary serve as additional segmentation-only targets, and DomiNite day-to-night serves as a second shift type, all without any target images during training. Metrics are target mIoU, depth RMSE and absolute relative error, and Δm against the pooled single-task models. Efficiency is reported in GPU-hours. Every number is averaged over three seeds and reported with confidence intervals.
The first decisive check is Phase 1's rank correlation. A high in-domain versus target correlation (we will treat a Spearman coefficient above 0.8 as high) falsifies the premise, and we stop to report the negative result. The second is the ablation against in-domain LDC-MTL. The held-out outer loop must improve target Δm while possibly losing slightly in-domain, which is the expected signature of tuning for transfer. The third is the budget-matched comparison with leave-one-domain-out grid search, which determines whether adaptivity adds anything beyond the selection signal. The fourth is whether per-task-per-domain weights beat per-task weights. This would show whether task and domain conflicts are entangled enough to need joint treatment.
References
- LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control
- Loss Function Learning for Domain Generalization by Implicit Gradient
- Episodic Training for Domain Generalization
- Multi-Task Learning for Dense Prediction Tasks: A Survey
- Domain Generalization for Semantic Segmentation: A Survey
- Semantic-Aware Domain Generalized Segmentation
- Domain Generalization via Gradient Surgery
- Gradient Matching for Domain Generalization
- Gradient Surgery for Multi-Task Learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models