Introduction
Multi-Task Learning (MTL) is a powerful paradigm designed to enhance the performance of multiple related tasks by learning them simultaneously. The core premise is that shared information can act as a regularizer, improving generalization across all tasks. However, the success of MTL hinges on selective sharing. If a model erroneously shares information between unrelated tasks, it suffers from "negative transfer," where joint learning performs worse than independent task learning.
Traditional MTL approaches often struggle with three main issues: the requirement to pre-specify the number of task clusters, over-sparsity (where models ignore redundant but relevant tasks), and an inability to exploit negative correlations. This paper introduces a novel framework to overcome these limitations.
The GAMTL Framework
The authors propose Group Adaptive Multi-Task Learning (GAMTL). In this framework, each task's parameter vector $w_i$ is represented as a linear combination of other tasks' parameters: $w_i \approx W_{-i}c_i$, where $C$ is a correlation matrix that captures task relationships.
The Trace Lasso Regularizer
The defining feature of GAMTL is the use of the Trace Lasso penalty. This regularizer acts as an adaptive bridge between the $\ell_1$ norm and the $\ell_2$ norm:
- Uncorrelated Tasks: When tasks are independent, Trace Lasso behaves like the $\ell_1$ norm, promoting sparsity and selecting only the most relevant predictors.
- Correlated Tasks: When tasks are highly related, it behaves like the $\ell_2$ norm, grouping them together to share information across the entire cluster.
This adaptivity allows the model to automatically discover task structures without requiring the user to define the number of clusters ($K$) beforehand. Furthermore, by allowing the correlation matrix to contain negative values, GAMTL can leverage negative correlations—where the success of one task predicts the failure of another—to improve learning.
Experimental Evaluation
The authors evaluated GAMTL against state-of-the-art baselines, including Adaptive Multi-Task Learning (AMTL) and Multi-Task Relationship Learning (MTRL), across synthetic and real-world datasets.
Structure Discovery and Clustering
In synthetic experiments designed to test group recovery, GAMTL demonstrated a superior ability to identify latent task structures. As shown in the comparison below, GAMTL successfully recovers the block-diagonal structure of task correlations, whereas baseline methods like AMTL produce sparse, disconnected results that fail to capture the group dynamics.
Performance on Real-World Benchmarks
- Multi-Domain Sentiment (MDS): On Amazon product reviews, GAMTL achieved a prediction error of 12.82%, a significant improvement over the Single Task Learning (STL) baseline of 28.77%.
- CIFAR-10: In image classification tasks, the model successfully grouped visually similar categories (e.g., different types of vehicles vs. different animals), leading to higher accuracy than competing MTL methods.
- School Dataset: Even in scenarios where task grouping is weak, GAMTL remained competitive with specialized low-rank methods, demonstrating its robustness.
Conclusion
GAMTL provides a robust solution for the selective sharing problem in Multi-Task Learning. By utilizing the Trace Lasso, it eliminates the need for manual cluster specification and effectively manages the trade-off between sparsity and grouping. While the optimization involves computationally intensive singular value decompositions (SVD), the model's ability to automatically discover complex task relationships marks a significant advancement in adaptive learning frameworks.
Summary updated