Adaptive Loss Balancing in Multi Task Learning

941 concepts613 papers9 proposals

When training a neural network to perform multiple tasks simultaneously—such as segmenting an image while estimating its depth, or fine-tuning a large language model on diverse instructions—we typically combine individual task losses into a single scalar objective:

$$L_{\text{total}} = \sum_{i} w_i L_i$$

In practice, setting these weights $w_i$ as static hyperparameters is a recipe for training failure. Different tasks naturally exhibit vastly different loss scales, optimization difficulties, and convergence rates. If a regression task has a loss magnitude orders of magnitude larger than a companion classification task, its gradients will dominate the shared parameters, causing the model to ignore the classification task entirely.

Even if we meticulously tune the initial weights, the optimization landscape shifts during training. A task that is easy to learn will quickly drop its loss, while a difficult task might stall. This imbalance can lead to catastrophic interference or early overfitting. To solve this, recent research focuses on dynamic task weighting: algorithms that automatically adjust $w_i$ or manipulate gradients in real time to ensure stable, cooperative multi-task convergence.

Balancing by Learning Pace

One intuitive way to balance multiple objectives is to ensure that all tasks progress at a similar relative speed. If Task A has already solved 90% of its objective while Task B has only solved 10%, the optimization framework should dynamically shift its focus toward Task B.

But how do we mathematically quantify a task's "pace"?

In their seminal work on gradient normalization, Zhao Chen et al. introduced GradNorm, which defines a task's relative training rate as the ratio of its current loss to its initial loss:

$$\tilde{L}_i(t) = \frac{L_i(t)}{L_i(0)}$$

By monitoring these ratios, the system can dynamically increase the weights of tasks that are converging slowly ($\tilde{L}_i$ remains high) and decrease the weights of tasks that are learning rapidly.

However, relying purely on training loss can be dangerous if a task begins to overfit. To protect the model from diverging on specific objectives, Hang Yu et al. proposed CoBa, which dynamically adjusts task weights based on the slopes of validation loss curves. Early in training, the algorithm prioritizes uniform learning speeds; later in training, if a task's validation loss begins to climb, the system automatically dials back its weight, acting as a targeted early-stopping mechanism for individual tasks within a shared architecture.

Alternatively, we can normalize the raw magnitudes of the losses themselves to prevent arbitrary scaling from dictating task importance. Anish Lakkapragada et al. demonstrated in Mitigating Negative Transfer in Multi-Task Learning with Exponential Moving Average Loss Weighting Strategies that scaling each task's loss by its own Exponential Moving Average (EMA) effectively flattens loss scale discrepancies, preventing tasks with naturally high loss values from drowning out subtler objectives.

Normalizing Gradient Magnitudes

While balancing loss curves provides a strong proxy for task progress, it ignores a fundamental reality of backpropagation: loss magnitude does not equal gradient magnitude. Two tasks can have identical loss values but produce gradients that differ by orders of magnitude due to the network's architecture or the mathematical formulation of the loss functions themselves.

This phenomenon is particularly acute in Physics-Informed Neural Networks (PINNs). Sifan Wang et al. analyzed this through the lens of the Neural Tangent Kernel (NTK) in When and why PINNs fail to train: A neural tangent kernel perspective. They showed that severe gradient imbalances between boundary condition losses and partial differential equation residuals cause standard gradient descent to fail completely. By using the trace of NTK sub-matrices to dynamically adjust loss weights, they proved that enforcing balanced gradient contributions across components guarantees stable convergence.

In computer vision and scene understanding, Ankit Jha et al. developed AdaMT-Net, which skips loss proxies entirely and dynamically scales weights based on the explicit magnitudes of the loss gradients during training. By ensuring that the shared encoder receives balanced gradient norms from semantic segmentation, depth estimation, and surface normal prediction, the model avoids developing a bias toward any single domain.

Without Balancing:
Task 1 Gradient: ====>
Task 2 Gradient: ========================> (Dominates shared parameters)

With Gradient Normalization:
Task 1 Gradient: ============>
Task 2 Gradient: ============> (Balanced updates)

Taking this a step further, Zhipeng Zhou et al. argue in Injecting Imbalance Sensitivity for Multi-Task Learning that gradient magnitude imbalance is often a more destructive hurdle than directional conflicts. They utilize Lagrangian multipliers to dynamically manage the trade-off between maximizing global average performance and maintaining Pareto optimality, ensuring that no single task's performance is sacrificed to benefit the majority.

Even if gradient magnitudes are perfectly matched, multi-task learning can still fail if the gradients point in opposite directions. When Task A requires a parameter to increase and Task B requires it to decrease, a standard gradient step averages them out, resulting in zero progress for both—a phenomenon known as negative transfer.

To resolve these directional stalemates, optimization strategies must look at the geometry of the task gradients. The authors of Conflict-Alleviated Gradient Descent for Adaptive Object Detection address this by monitoring the alignment between domain adaptation and object detection tasks. When gradients conflict, the algorithm projects them onto each other's normal planes, effectively filtering out the destructive components of the updates so that the tasks work cooperatively rather than destructively.

Task A Gradient (↑) vs Task B Gradient (↓) 
  --> Static Sum = Zero Progress
  --> Conflict Alleviated Projection = Cooperative Lateral Progress (→)

Framing this challenge through the lens of Multi-Objective Optimization (MOO), Lisha Chen demonstrates in M2ASR: Multilingual Multi-task Automatic Speech Recognition via Multi-objective Optimization that dynamically adjusting weights based on the real-time state of gradient conflict allows models to navigate highly complex optimization landscapes. Similarly, Shaocong Long leverages a momentum-based inverse training rate in Diverse Target and Contribution Scheduling for Domain Generalization to dynamically adjust domain contributions, preventing the model from overfitting to easy environments while actively resolving gradient conflicts with harder, out-of-distribution domains.

By shifting from static, manual hyperparameter tuning to dynamic loss and gradient balancing, these frameworks transform multi-task training from a fragile tug-of-war into a synchronized, cooperative optimization process.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_task_weighting_d05b0b3d,
  title = {dynamic task weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-task-weighting/rdr_d05b0b3df03d42a7b82efc803ad7f3ce},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_task_weighting_d05b0b3d_v1,
  title = {dynamic task weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-task-weighting/rdr_d05b0b3df03d42a7b82efc803ad7f3ce/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake