A Simple General Approach to Balance Task Difficulty in Multi-Task Learning

Sicong Liang, Yu ZhangarXiv 2020 · OpenReview

Abstract

In multi-task learning, difficulty levels of different tasks are varying. There are many works to handle this situation and we classify them into five categories, including the direct sum approach, the weighted sum approach, the maximum approach, the curriculum learning approach, and the multi-objective optimization approach. Those approaches have their own limitations, for example, using manually designed rules to update task weights, non-smooth objective function, and failing to incorporate other functions than training losses. In this paper, to alleviate those limitations, we propose a Balanced Multi-Task Learning (BMTL) framework. Different from existing studies which rely on task weighting, the BMTL framework proposes to transform the training loss of each task to balance difficulty levels among tasks based on an intuitive idea that tasks with larger training losses will receive more attention during the optimization procedure. We analyze the transformation function and derive necessary conditions. The proposed BMTL framework is very simple and it can be combined with most multi-task learning models. Empirical studies show the state-of-the-art performance of the proposed BMTL framework.

Introduction

Multi-Task Learning (MTL) is a paradigm designed to improve the generalization performance of multiple related tasks by learning them simultaneously. A core challenge in MTL is the inherent imbalance in task difficulty. When tasks are learned together, a "hard" task may be overshadowed by an "easy" one, or the model may struggle to converge if it focuses too heavily on a single complex objective. Traditional methods, such as manually tuning task weights or using curriculum learning schedules, often require significant domain expertise and trial-and-error.

The BMTL Framework

The proposed Balanced Multi-Task Learning (BMTL) framework introduces an elegant mathematical solution to this balancing act. Instead of minimizing the simple sum of losses, BMTL minimizes a sum of transformed losses:

$$\min \sum h(L_i)$$

where $h(\cdot)$ is a monotonically increasing and strongly convex function. The brilliance of this approach lies in the gradient calculation. By the chain rule, the gradient of the transformed loss is $\nabla h(L_i) = h'(L_i) \nabla L_i$. Here, the derivative $h'(L_i)$ acts as a dynamic, self-adjusting weight. When a task has a high loss (indicating it is "hard" or currently under-learned), $h'(L_i)$ becomes large, forcing the optimizer to prioritize that task. As the loss decreases, the weight naturally diminishes, allowing the model to shift focus to other struggling tasks.

Experimental Evaluation

The researchers evaluated BMTL across several diverse benchmarks, including computer vision datasets like Office-31, Office-Home, and ImageCLEF, as well as the SARCOS robotics dataset for inverse dynamics. They compared BMTL against seven established strategies, including Multi-Objective Gradient Descent (MGDA), Dynamic Weight Averaging (DWA), and Curriculum Learning (CL).

Accuracy comparison across different training proportions for various MTL methods. BMTL (yellow star) consistently outperforms baselines.

Key Results and Insights

As shown in the experimental results, BMTL (represented by the yellow star line) consistently achieves the highest accuracy across varying training proportions.

  • Self-Correction: The framework is inherently self-balancing, requiring no manual intervention to adjust weights as training progresses.
  • Smooth Optimization: Unlike the "Maximum" approach, which can lead to non-smooth objective functions, BMTL uses smooth convex transformations (like the exponential function) that are easily handled by standard optimizers like Adam or SGD.
  • Robustness: The performance gains are consistent regardless of whether the training data constitutes 50%, 60%, or 70% of the dataset, demonstrating the method's stability.
Performance of BMTL compared to other methods in an alternative task configuration, showing consistent superiority.

Conclusion

BMTL provides a simple, theoretically grounded, and highly effective framework for balancing task difficulty in multi-task learning. Its "plug-and-play" nature allows it to be integrated into existing deep MTL architectures with minimal code changes, making it a powerful tool for practical machine learning applications where task imbalances are common.

Summary updated

Abstract

In multi-task learning, difficulty levels of different tasks are varying. There are many works to handle this situation and we classify them into five categories, including the direct sum approach, the weighted sum approach, the maximum approach, the curriculum learning approach, and the multi-objective optimization approach. Those approaches have their own limitations, for example, using manually designed rules to update task weights, non-smooth objective function, and failing to incorporate other functions than training losses. In this paper, to alleviate those limitations, we propose a Balanced Multi-Task Learning (BMTL) framework. Different from existing studies which rely on task weighting, the BMTL framework proposes to transform the training loss of each task to balance difficulty levels among tasks based on an intuitive idea that tasks with larger training losses will receive more attention during the optimization procedure. We analyze the transformation function and derive necessary conditions. The proposed BMTL framework is very simple and it can be combined with most multi-task learning models. Empirical studies show the state-of-the-art performance of the proposed BMTL framework.
BibTeX
@misc{liang2020simple,
  author = {Sicong Liang and Yu Zhang},
  title = {{A Simple General Approach to Balance Task Difficulty in Multi-Task Learning}},
  year = {2020},
  eprint = {2002.04792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.04792},
}
Figure