Balance Before You Backpropagate: Multi-Task Loss Weights as Per-Task Loss Scales in Low Precision

Multi-task learning trains one shared network on several objectives at once, for example segmentation, depth and surface normals from the same image. A large body of work adjusts how much each task contributes to every update. These methods rescale, project or reweight each task's gradient after it has been computed, and they assume that gradient is faithful. In half precision (FP16), or in 8-bit floating point with a similarly narrow exponent range, that assumption fails first for the tasks these methods most want to protect. Mixed-precision training multiplies the loss by one global "loss scale" so that small gradients stay representable. When task gradients differ by orders of magnitude, a single scale sized to avoid overflow in the loudest task lets the quietest task's gradient round to zero during the backward pass. Any balancing applied afterward then amplifies whatever survived the rounding.

This note proposes to move balancing to the other side of the backward pass. Before backpropagating, we compute a per-task multiplier in log space from running estimates of each task's gradient norm and apply it to that task's loss. One backward pass then carries every task's gradient at a similar, representable magnitude. The claim is that this multiplier does two jobs at once. It is the magnitude-balancing weight that multi-task methods such as Dual-balancing for multi-task learning already found beneficial, and it is also the per-task loss scale that low precision requires. If the claim holds, magnitude-balanced multi-task training becomes possible in FP16 or FP8 at the cost of one backward pass, with quality matching per-task-backward FP32 methods.

Core Claim

Our hypothesis is that in narrow-exponent formats, balancing gradients after backpropagation and weighting losses before it are not interchangeable. Post-hoc gradient balancing degrades as precision drops, and the degradation is concentrated in tasks with small gradients. Pre-backward, log-domain loss weighting recovers FP32 quality. The mechanism is simple. Let $g_k$ be task $k$'s gradient, $\hat{\ell}_k$ a moving average of $\log \|g_k\|$ measured on the previous steps, and $\tau$ a target log-magnitude chosen to sit safely inside the format's range. We set

$$\log w_k = \mathrm{clip}\big(\tau - \hat{\ell}_k,\; a,\; b\big), \qquad \mathcal{L} = \sum_{k=1}^{K} w_k \mathcal{L}_k,$$

where $w_k$ is task $k$'s loss multiplier and $\mathcal{L}_k$ its loss. The bounds $a$ and $b$ are derived from the format's smallest and largest representable exponents, with a margin for the spread of values inside a tensor. The weights are computed and stored as logarithms, so they never overflow or underflow themselves.

Task-specific heads receive only their own task's gradient, so their numerical scale can be divided back out exactly in the FP32 master weights. This lets heads follow any optimization weighting we choose. Shared parameters receive the weighted sum. For them, the numerically safe weighting and the magnitude-balanced weighting are the same thing, and that coincidence is what the method relies on.

The same construction handles a second failure. The log-loss transform used by Dual-balancing, and the $\alpha$-fair utilities of Fair Resource Allocation in Multi-Task Learning, give task $k$ an effective weight of $1/\mathcal{L}_k$ or $\mathcal{L}_k^{-\alpha}$. That weight overflows as a task's loss approaches zero. Computed as $-\alpha \log \mathcal{L}_k$ and clipped into the same window, it stays bounded.

Relation to Prior Work

Three lines of multi-task work supply the weighting schemes we intend to make precision-safe.

All of these operate on gradients after backpropagation, all were evaluated in full precision, and all but pure loss weighting require one backward pass per task. We keep their weighting logic. What changes is where the weight enters the computation.

Two low-precision papers supply the numerical constraint and part of the method.

  • Reducing Underflow in Mixed Precision Training by Gradient Scaling shows that one global loss scale fails when different branches have divergent gradient distributions. It fixes this with analytical scales computed per gradient tensor, evaluated only on single-task networks. A per-tensor scale cannot separate tasks inside a shared tensor, because that tensor already holds their sum. Our per-task scale acts on the loss, before the sum is formed.
  • Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision shows that FP16 reinforcement learning needs specific numerical fixes: an Adam variant that avoids squaring gradients, and Kahan (compensated) summation. We adopt these unchanged for the multi-task RL experiments; they are orthogonal to our contribution.

Log-number representations as in Learning in Integer Latent Variable Models with Nested Automatic Differentiation and the log-space mirror updates of SCENT (A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimization) motivate computing the weights in log space. We deliberately do not carry whole gradients in log form. Their papers document the arithmetic overhead that doing so would impose.

Research Program

The work proceeds in three stages.

Diagnosis. On each benchmark we run FP32 multi-task training and, at matched checkpoints, recompute each task's backward pass in FP16 and in emulated FP8 (E5M2, the gradient format) under standard dynamic loss scaling. We then measure, per task and per layer, the fraction of gradient entries that are nonzero in FP32 but zero at low precision. We also measure the cosine similarity between the low-precision and FP32 gradients, and the bias this introduces into the gradient norms that Dual-balancing and FairGrad use as inputs. The prediction is that underflow concentrates in tasks with small losses. QM9's eleven regression targets of very different scale and CelebA's forty binary attributes are where we expect it to be most visible.

Method. We implement the log-domain pre-backward weighting above for linear scalarization, Dual-balancing and FairGrad. The gradient-norm estimates come from the task-specific head gradients and the gradient at the last shared layer. These are available from the single backward pass, so no extra passes are required.

Separating two failure modes. BF16 has FP32's exponent range but only 7 mantissa bits. It therefore isolates a different failure: small task contributions being absorbed when they are added to large ones in a shared tensor. We test whether pre-backward balancing also reduces this absorption, which it should, because the summed terms have similar magnitudes.

Baselines

Naive reference. Linear scalarization, meaning an unweighted sum of task losses, trained in FP16 with a single dynamic loss scale. This is what a practitioner gets by default from standard mixed-precision tooling. An FP32 run of the same model serves as the ceiling.

Strongest existing method. FP32 Dual-balancing and FP32 FairGrad, run with K backward passes as published. For reference, on NYUv2 FairGrad reports a performance drop relative to single-task models of −4.66% (Fair Resource Allocation in Multi-Task Learning) and CAGrad reports 0.20% (Conflict-Averse Gradient Descent for Multi-task learning). On CelebA, FairGrad reports a 0.37% drop and a mean rank of 4.95. On Meta-World MT50, CAGrad reports a success rate of 0.52 and PCGrad 0.50. Dual-balancing reports no scalar values on NYUv2, Cityscapes or QM9 in the file ("no reported number found on Lacuna" for those), so we will rerun it ourselves.

Strongest low-precision competitor. The same K-backward Dual-balancing and FairGrad in FP16, each task with its own independent dynamic loss scale, with aggregation done in FP32. This is the natural patch a skeptic would write. We also run the per-tensor gradient scaling of Zhao et al. on top of a single-backward linear sum.

Fairness of comparison. All runs share architectures, data splits, step counts, and a hyperparameter search budget of equal size per method. Wall-clock time and peak memory are reported alongside accuracy.

What counts as success. Beating the naive reference means closing most of its gap to FP32. Beating the strongest methods means the single-backward FP16 or FP8 method lands within seed noise of FP32 Dual-balancing and FairGrad on the Δm metric (the average per-task performance drop relative to single-task models) while using roughly 1/K of their backward compute.

Ablations. The key ablation holds the weights fixed and moves them from after the backward pass to before it. This isolates the claim that placement, not the weights themselves, is what matters. Two further ablations compute the weights in linear rather than log space, and remove the representable-range clip.

Why not just…?

Why not just use BF16, which has FP32's exponent range? This is a genuine threat. If BF16 matches FP32 on every benchmark, our range-based argument applies only to FP16 and FP8 hardware, and we will say so. The BF16 arm is designed to settle this. It also tests whether the separate absorption problem, small task gradients lost when added to large ones in a shared tensor, harms multi-task training, which exponent range alone does not fix.

Why not just give each task its own dynamic loss scale and do surgery in FP32? This plausibly works as well on quality, so we run it as the strongest baseline rather than argue against it. It has two costs. It needs K backward passes, which becomes prohibitive for CelebA's forty tasks. And any single task's overflow forces the whole step to be skipped, so skipped steps become more frequent as K grows. We will report skip rates. The contribution stands only if the single-backward method matches this baseline's quality at lower compute.

Why not just reformulate PCGrad or CAGrad projections in log space, or add an epsilon to the denominators? The file's gap analysis locates the underflow in the projection denominators. But once per-task gradients are cast to FP32 for aggregation, norms and inner products can be computed stably without logarithms. Our diagnosis stage tests the alternative view: that the information is already lost upstream, in the low-precision backward pass, where no downstream reformulation can recover it. If the measured underflow is instead concentrated in the projection step, the proposal is wrong, and log-space surgery would be the right target.

Why not just use per-tensor gradient scaling? Per-tensor scaling (Zhao et al.) chooses one scale for each shared tensor after the task gradients have already been summed. It therefore cannot rescue a quiet task inside that tensor. We run it both alone and combined with our weighting to test whether the two effects add.

Decisive Experiments

We run supervised multi-task training on NYUv2, Cityscapes, QM9 and CelebA with the backbones used by Dual-balancing and FairGrad. Multi-task RL runs on Meta-World MT10 and MT50, using multi-task Soft Actor-Critic with the half-precision fixes of Björck et al. applied throughout. Each setting is run in FP32, BF16, FP16 and emulated FP8, with five seeds.

Three results would decide the question. First, the diagnosis must show underflow concentrated in small-gradient tasks, and that concentration must predict the per-task accuracy loss of post-hoc balancing at low precision. If it does not, the premise fails. Second, the placement ablation must show that the same weights help when applied before the backward pass and not when applied after. Third, on MT50 the single-backward FP16 agent should approach CAGrad's reported 0.52 success rate while retaining the memory and speed benefits Björck et al. reported for FP16 training: about 1.8–1.9x lower memory and more than 2x speedup on wide networks. A null result on MT50 alongside positive supervised results would itself be informative. It would suggest that in reinforcement learning, moving targets rather than exponent range are the binding constraint.