In deep learning, we frequently want a single neural network to perform multiple tasks simultaneously. For example, an autonomous driving model might need to detect pedestrians, segment lane lines, and estimate depth all from a single camera feed.
To train such a network, we must combine multiple independent, task-specific loss functions into a single scalar objective that a standard gradient descent optimizer can minimize. This process is known as loss scalarization. The most common approach is a weighted linear combination:
$$L_{\text{total}} = \sum_{i} w_i L_i$$
Choosing the weights $w_i$ is a deceptively difficult challenge. If the weights are poorly tuned, the gradients of one dominant task can overwhelm the others, causing the network to neglect secondary objectives or suffer from catastrophic gradient interference.
The Baseline: Unitary Scalarization
The simplest approach to multi-task optimization is unitary scalarization, where all task weights are set to one ($w_i = 1$). In this setup, we simply sum the individual losses.
While this sounds naive, Antonio Demontis et al. demonstrate in In Defense of the Unitary Scalarization for Deep Multi-Task Learning that unitary scalarization is remarkably robust. When paired with standard regularization techniques like dropout and weight decay, it can match or outperform highly complex, specialized multi-task optimizers without the computational overhead of calculating per-task gradients. Similarly, Li Zhang et al. successfully leverage an unweighted sum of losses in Multi-Label Transfer Learning for Multi-Relational Semantic Similarity, operating under the assumption that a sufficiently versatile shared encoder can naturally satisfy multiple objectives without explicit balancing.
However, unitary scalarization encounters a fundamental mathematical barrier when tasks are highly heterogeneous. If Task A computes a Mean Squared Error loss (which can scale to large values) and Task B computes a Cross-Entropy loss (which is strictly bounded), the gradient magnitudes will differ by orders of magnitude. The optimizer will naturally prioritize the task with the larger loss magnitude, effectively turning the multi-task network into a single-task network.
Static and Scheduled Weighting
To prevent a single task from dominating the gradient space, researchers introduce hyperparameters to manually scale each loss contribution.
In A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding, Houfeng Wang et al. utilize a tunable weight parameter to balance error signals between disparate natural language tasks, preventing error propagation during joint training. This explicit balancing is also crucial when integrating auxiliary tasks designed solely to guide the primary model. As shown by Kai-Bin Huang et al. in Applying Multi-Task Deep Learning Methods in Electricity Load Forecasting Using Meteorological Factors, assigning specific weights to auxiliary predictions allows the model to extract useful shared representations without letting secondary objectives degrade primary task performance.
Scheduled Weighting
The optimal weight for a task is not necessarily static throughout the training cycle. At the beginning of training, a model might need to focus on easy, foundational features before tackling complex, fine-grained details.
To capture this intuition, Chenliang Li et al. introduce a scheduled loss weighting mechanism in Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction. They apply a linear decay schedule to an auxiliary extraction loss:
Loss Weight
^
1.0|-----------\
| \ Auxiliary Extraction Loss (Decays)
| \
0.0+--------------> Time
Early in training, the high auxiliary weight forces the encoder to prioritize evidence localization. As training progresses and the model masters localization, the auxiliary weight decays to zero, shifting the optimization focus entirely toward natural language generation.
Dynamic and Uncertainty-Driven Weighting
Manual hyperparameter tuning becomes impossible when a model scales to dozens of tasks. For instance, Ethan M. Rudd demonstrates in MOON: A Mixed Objective Optimization Network for the Recognition of Facial Attributes that a mixed-objective network optimizing for dozens of binary facial attributes simultaneously requires a dynamic, adaptive loss function to handle the structural imbalances of multi-label datasets.
A elegant solution to this problem is to let the network learn its own loss weights dynamically during training.
Homoscedastic Uncertainty as a Weighting Metric
The authors of the seminal paper Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics derived a principled, probabilistic framework for multi-task loss balancing. They treat each task's output as a probability distribution and maximize the log-likelihood of the joint model.
For a regression task with model output $y$ and ground truth $x$, the loss function conditioned on the task's homoscedastic uncertainty (task-dependent variance $\sigma^2$) is formulated as:
$$L(W, \sigma) = \frac{1}{2\sigma^2} ||y - x||^2 + \log \sigma$$
This formulation reveals a beautiful intuition:
- The Attenuator: The term $\frac{1}{2\sigma^2}$ acts as an adaptive weight. If a task is highly noisy or difficult, its variance $\sigma^2$ increases, which automatically down-weights the task's loss so it doesn't corrupt the shared features.
- The Regularizer: The term $\log \sigma$ acts as a penalty. Without it, the network could simply set $\sigma \to \infty$ for every task to reduce the total loss to zero. This term forces the network to keep its uncertainty estimates grounded.
Building on the concept of adaptive optimization, Shiping Ma et al. implement learnable weights to balance highly sensitive geometric and orientation losses in Learning Rotated Inscribed Ellipse for Oriented Object Detection in Remote Sensing Images. To bypass the computational overhead of updating these weights iteratively via backpropagation, Dingwen Li et al. introduce a closed-form analytic solution for balancing task-specific losses in DeepAlerts: Deep Learning Based Multi-Horizon Alerts for Clinical Deterioration on Oncology Hospital Wards, providing a highly efficient alternative for complex clinical forecasting systems.
Normalizing Heterogeneous Loss Spaces
When a multi-task network combines structurally diverse loss functions—such as cross-entropy for classification, cosine similarity for geometry, and L1 loss for regression—the absolute values of the losses are fundamentally mismatched. Steven Hickson et al. highlight this challenge in Sharing Decoders: Network Fission for Multi-task Pixel Prediction, where a shared decoder must simultaneously output categorical semantics and continuous geometric depth maps.
A major source of gradient imbalance in sequence-to-sequence multi-task networks is variation in sequence length. If Task A processes long text sequences and Task B processes short sequences, the cumulative loss of Task A will naturally dwarf Task B simply because it sums over more tokens.
To resolve this, Tejas Srinivasan et al. introduce a sequence-length normalization strategy in Multitask Learning For Different Subword Segmentations In Neural Machine Translation. Before aggregating the losses from multiple decoders into a single scalar sum, each individual loss is normalized by its respective sequence length:
$$L_{\text{normalized}} = \frac{1}{N} \sum_{t=1}^{N} L_t$$
By scaling the loss relative to the sequence length $N$, the gradient contribution per token remains uniform across all tasks. This simple mathematical normalization prevents decoders with longer target granularities from disproportionately pulling the shared encoder parameters toward their objectives, ensuring stable, collaborative multi-task learning.