Introduction
Multi-task learning (MTL) is a powerful paradigm where a single model is trained to perform several tasks simultaneously, such as identifying facial expressions while detecting specific facial attributes. However, MTL frequently suffers from negative transfer, a phenomenon where the performance of individual tasks degrades compared to single-task models. This often occurs when loss values across different tasks have vastly different scales or convergence speeds, leading the optimizer to prioritize tasks with larger loss magnitudes.
Existing solutions, such as GradNorm or Uncertainty Weighting (UW), attempt to balance these losses using complex gradient optimization or probabilistic modeling. However, these methods can be computationally expensive or prone to "volatile spikes" in weighting that destabilize training. This paper proposes a simple yet effective solution: scaling each task's loss by its own Exponential Moving Average (EMA) to normalize losses to a similar scale.
Methodology
The authors introduce three strategies centered on scaling losses by their observed magnitudes:
- Exponential Moving Average (EMA): The core contribution involves calculating a running average of each task's loss ($\tilde{L}_k$) and setting the task weight ($\lambda_k$) to the reciprocal of that average ($1/\tilde{L}_k$). This effectively scales every task's loss to a magnitude of approximately 1.0.
- Rated Exponential Moving Average (REMA): This variant combines EMA scaling with a "training rate" component, assigning extra weight to tasks that converge slowly (where the ratio of current loss to previous loss is high).
- Dynamic Weighted Exponential Moving Average (DWEMA): A variation of the Dynamic Weight Average (DWA) method, DWEMA first scales losses by their EMA to prevent any single task from dominating the rate calculation.
The shift from learning weights via backpropagation to calculating them directly from loss history offers improved stability and implementation simplicity.
Experimental Setup
The proposed methods were evaluated on three major facial analysis benchmarks:
- CelebA: 40 binary facial attributes (e.g., "Smiling," "Wearing Glasses").
- AffWild2: A complex dataset involving Action Units (AU), Emotion classification, and Valence & Arousal (VA) estimation.
- AffectNet: Focused on Emotion and Valence & Arousal.
These datasets require the model to extract diverse features from the same input image, making them ideal for testing MTL strategies.
Results and Analysis
The EMA-based strategies consistently outperformed baselines and established methods like GradNorm and UW across all datasets.
Performance on AffWild2 and AffectNet
On the AffWild2 dataset, the EMA method achieved the highest overall score of 0.800, significantly outperforming the single-task average of 0.699. For AffectNet, EMA reached an overall score of 0.898, compared to the baseline's 0.853.
Performance on CelebA
On the CelebA dataset, REMA and DWEMA achieved a performance score of 0.788, outperforming both the Baseline (0.772) and GradNorm (0.771).
Key Insights
- Simplicity and Stability: Normalizing tasks to the same numerical scale is often more critical than sophisticated balancing. The use of a $\beta$ hyperparameter in EMA calculations provides a buffer that prevents volatile spikes in loss coefficients.
- Optimization Efficiency: The authors note that Uncertainty Weighting (UW) often converges to weights roughly equal to $1/L$. By setting weights to $1/EMA(L)$ directly, the proposed method bypasses the noisy optimization process inherent in UW.
Conclusion
This paper demonstrates that simple EMA-based loss scaling can effectively mitigate negative transfer in multi-task learning. While the methods require tuning the $\beta$ hyperparameter, their low computational overhead makes them ideal for edge device applications. Future work should explore the sensitivity of these hyperparameters and the applicability of EMA scaling to other domains such as Natural Language Processing and Autonomous Driving.
Summary updated