Multi-task networks share one backbone across several objectives. For example, one image network might predict segmentation, depth and surface orientation at once. Popular fixes for tasks interfering with one another all rest on a few statistics computed from minibatch gradients. They rescale each task's loss so that gradient sizes match, or they detect "conflict," meaning two task gradients point in opposing directions, and remove it. These statistics are sample averages. A separate line of theory shows that signals inside deep networks become increasingly heavy-tailed as depth grows, meaning rare, extreme values dominate. The same holds for the noise in stochastic gradients. When the tails are heavy enough, averages over a minibatch converge very slowly, and a single outlier batch can flip the measured sign of a gradient inner product. This proposal asks a simple question: how much of the "conflict" that multi-task methods react to is real disagreement between tasks, and how much is heavy-tail sampling noise that gets worse deeper in the backbone?
The practical stakes are direct. If balancing weights and conflict decisions are driven by a few extreme batches, then more elaborate conflict-resolution machinery is partly treating a measurement problem. The problem would then be better addressed by measuring more carefully, layer by layer, using estimators that tolerate heavy tails. If the answer is "mostly real," that is also worth knowing, because it would support the more expensive geometric methods.
Core Claim
We claim that per-task gradient statistics in shared backbones become heavy-tailed at a rate that depends on depth. Under those tails, the gradient norms used by loss-balancing methods and the gradient cosines used by conflict-removal methods are unreliable when computed from single minibatches. Replacing them with robust estimates computed per layer block, such as a median of several microbatch estimates, should do three things. It should reduce the measured rate of conflict at the heavy-tailed depths. It should stabilize loss weights. And it should recover a meaningful share of the gains that more complex methods attribute to resolving directional conflict.
What would be new is threefold: treating the balancing statistics themselves as estimators under heavy tails, measuring their tail index by task and by depth, and choosing where and how to measure conflict based on that diagnostic. None of the cited multi-task methods does any of these.
The core quantitative intuition comes from the generalized central limit theorem. Suppose per-sample gradient components have tail index $\alpha$, meaning $P(|g|>t)$ decays roughly like $t^{-\alpha}$, with $1<\alpha<2$. Then the error of a $B$-sample mean shrinks like
$$\left|\bar g_B - \mathbb{E}[g]\right| \sim B^{1/\alpha - 1},$$
rather than $B^{-1/2}$. At $\alpha = 1.55$ this is roughly $B^{-0.35}$, so quadrupling the batch buys much less than it would under Gaussian noise. A median-of-means estimate splits the batch into $m$ microbatches with means $\bar g^{(1)},\dots,\bar g^{(m)}$ and takes
$$\hat g = \operatorname{median}\left(\bar g^{(1)},\dots,\bar g^{(m)}\right),$$
applied to norms and cosines. It is known to concentrate much better than the plain mean under heavy tails. We use it as the default robust estimator and compare it against trimmed means.
Grounding in Prior Work
Several papers motivate the claim that tails grow with depth. Understanding Priors in Bayesian Neural Networks at the Unit Level proves that hidden units in Gaussian-weight networks are sub-Weibull with tail parameter $\theta=\ell/2$ at layer $\ell$, so each layer is heavier-tailed than the last. Bayesian neural network unit priors and generalized Weibull-tail property tightens this to two-sided bounds. Bayesian neural networks increasingly sparsify their units with depth shows that batch and layer normalization do not change the tail parameter. All three describe networks at initialization, not during training.
The closest evidence about training comes from Multiplicative Noise and Heavy Tails in Stochastic Optimization. It models optimizers as random multiplicative recurrences and reports that the tail exponent of parameter fluctuations on CIFAR-10 falls from about 6.1 for ResNet-9 to about 1.55 for ResNet-68. That result concerns single-task training and the stationary distribution of parameters, not per-sample task gradients. Transferring it to gradient statistics is therefore a hypothesis we must measure, not assume.
One result constrains the picture. Bayesian Neural Network Priors Revisited finds that trained ResNet weights are heaviest-tailed in early layers and closer to Gaussian in deep layers, the opposite of the depth ordering predicted for units at initialization. We therefore do not presuppose that deeper means heavier. We measure the tail index per block, per task and per training phase. Where the tails actually sit is itself a result.
On the multi-task side, GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks adjusts scalar loss weights so that per-task gradient norms on one shared layer, usually the last, match targets set by relative training rates. RotoGrad: Gradient Homogenization in Multitask Learning learns task-specific rotations of the final shared representation to align gradient directions, and it compares against projection-based conflict removal (PCGrad). Both methods supply the machinery we reuse. Both treat per-step gradient statistics at a single layer as trustworthy, and that is the assumption we test.
Research Program
The first stage is purely diagnostic. For shared backbones of increasing depth, we log per-task gradients at every residual block for a subset of training steps. We store per-sample or per-microbatch gradients for a fixed probe set. From these we estimate three things. The first is the tail index of per-task gradient norms and of individual coordinates, using a Hill estimator with a sensitivity analysis over the cutoff. The second is the spread of the per-step estimates of balancing norm and pairwise cosine. The third is a "conflict reliability" score: the probability that the sign of a task-pair cosine measured on one minibatch agrees with the sign measured on a reference batch 32 times larger. If conflict is real, reliability should be high wherever conflict is frequent. If it is mostly noise, reliability should fall toward one half at blocks with low tail index.
The second stage is an intervention. We keep GradNorm, PCGrad and RotoGrad unchanged, except that every statistic they compute from gradients is replaced by its median-of-means counterpart over $m$ microbatches of the same total batch. The parameter update itself is left unclipped and unmodified. A second variant, which we call depth-selective, applies the balancing statistic at the block whose tail index is highest, the most reliable block, rather than always at the last shared layer. This tests whether "where to measure" matters as much as "how to measure."
The third stage links the two. Across depths and tasks, we regress the gain from robust estimation on the measured tail index. The prediction is that gains appear where $\alpha$ is low and vanish where gradients are near-Gaussian.
Baselines
The naive reference points are equal-weighted multi-task training with vanilla summed losses and $K$ separately trained single-task networks. Together they bound the achievable benefit from sharing. The strongest existing method is RotoGrad. On NYUv2 surface normals it reports a median angle error of 20.80 against 26.16 for vanilla multi-task training (RotoGrad). GradNorm is the direct target of our loss-balancing intervention and reports roughly a 5% depth-error improvement over standard multi-task training on NYUv2, at about 5% extra training time (GradNorm). PCGrad is the direct target of our conflict-detection intervention. For CelebA, Multi-MNIST, CIFAR-10-as-10-tasks and the facial landmark set, the file lists the benchmarks but no numbers, so there is no reported result found on Lacuna for those baselines, and we rerun all of them ourselves.
Fairness is enforced in four ways. Every method sees the same data, backbone, augmentation, total batch size and number of epochs. Each receives the same hyperparameter search budget, with learning rate and method-specific coefficients tuned over identical grids on a held-out split. Our robust variants split the same batch into microbatches rather than drawing more data. We report wall-clock cost alongside accuracy.
The key ablation isolates robust estimation from everything else. We run robust PCGrad against PCGrad with the statistics computed on the plain mean of the same microbatches. The only difference between the two is the median.
Beating the baselines means two things. A robust GradNorm or PCGrad variant should improve on its plain counterpart by more than seed-to-seed variation over at least five seeds. And robust PCGrad should close a substantial part of the gap between PCGrad and RotoGrad, for instance on NYUv2 normals, while keeping PCGrad's lower cost. Matching RotoGrad outright would be the strong result. Improving RotoGrad itself through robust statistics would show that the effect is complementary.
Why not just…?
Why not just use a larger batch? If the noise were Gaussian, this would be the obvious fix. Under tail index $\alpha$ below 2, the error of the batch mean shrinks only like $B^{1/\alpha-1}$, so at the depths where tails are heavy it buys little. We test this directly with a batch-size sweep at fixed epochs, measuring conflict reliability as a function of $B$ at each block. If reliability rises at the Gaussian rate everywhere, our premise fails and larger batches are the right answer.
Why not just clip gradients? Clipping is a real threat, because it also suppresses extreme batches. It differs in that it changes the update itself, not just the decision statistics, and it needs a threshold that must itself be tuned per depth. We include tuned global and per-block clipping as baselines, combined with each multi-task method. If clipped PCGrad matches robust PCGrad, we will report that the practical fix is clipping and that our contribution reduces to the diagnostic.
Why not just smooth the statistics with a moving average? GradNorm already smooths implicitly by updating its weights through gradient descent. A moving average is still a mean and inherits the slow convergence of means under heavy tails, whereas the median does not. We run exponential-moving-average versions of the norm and cosine statistics with tuned decay as a baseline. Their stability of loss weights and their final metrics are the comparison.
Why not just rely on normalization layers to remove the tails? Bayesian neural networks increasingly sparsify their units with depth shows that normalization rescales variance but leaves the tail parameter unchanged, at least at initialization. Our diagnostic uses normalized ResNets throughout, so the measured tail indices answer this empirically for trained networks.
Why not just move RotoGrad's rotations deeper into the backbone? That is RotoGrad's own suggested extension, and it is orthogonal to this question. If deep-layer conflicts are mostly noise, deeper rotations would chase spurious targets. We run a two-block rotation variant with and without robust statistics to test exactly that interaction.
Proposed Experiments
The depth sweep uses CIFAR-10 recast as ten binary tasks, the multi-task form used by RotoGrad. It spans Wide ResNet depths matching the single-task sweep in Multiplicative Noise and Heavy Tails in Stochastic Optimization, so our per-task gradient tail indices can be set beside its reported parameter tail exponents of 6.1 and 1.55. Multi-MNIST, where tasks are known to conflict, and CelebA with 40 tasks test the method on genuine and many-task conflict. NYUv2, with segmentation (mean IoU), depth (absolute error) and surface normals (median angle), is the headline benchmark on ResNet backbones of two depths. We report per-task metrics, the average relative change against single-task networks, loss-weight trajectories, and conflict reliability per block.
Three checks decide the outcome. First, conflict reliability must fall measurably at low-$\alpha$ blocks. If per-minibatch cosine signs agree with large-batch signs everywhere, the core claim is false. Second, gains from robust estimation must track measured tail index across depths and tasks. Uniform gains would point to generic variance reduction rather than tails. Third, robust PCGrad must beat tuned clipping and moving-average baselines, and not merely match them, before we claim robust estimation as a method rather than a diagnosis.
References
- Understanding Priors in Bayesian Neural Networks at the Unit Level
- Bayesian neural network unit priors and generalized Weibull-tail property
- Bayesian neural networks increasingly sparsify their units with depth
- Multiplicative Noise and Heavy Tails in Stochastic Optimization
- Bayesian Neural Network Priors Revisited
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
- RotoGrad: Gradient Homogenization in Multitask Learning