Adaptive Group Sparse Multi-task Learning via Trace Lasso

Sulin Liu, Sinno Jialin Pan

Introduction

Multi-Task Learning (MTL) is a powerful paradigm designed to enhance the performance of multiple related tasks by learning them simultaneously. The core premise is that shared information can act as a regularizer, improving generalization across all tasks. However, the success of MTL hinges on selective sharing. If a model erroneously shares information between unrelated tasks, it suffers from "negative transfer," where joint learning performs worse than independent task learning.

Traditional MTL approaches often struggle with three main issues: the requirement to pre-specify the number of task clusters, over-sparsity (where models ignore redundant but relevant tasks), and an inability to exploit negative correlations. This paper introduces a novel framework to overcome these limitations.

The GAMTL Framework

The authors propose Group Adaptive Multi-Task Learning (GAMTL). In this framework, each task's parameter vector $w_i$ is represented as a linear combination of other tasks' parameters: $w_i \approx W_{-i}c_i$, where $C$ is a correlation matrix that captures task relationships.

The Trace Lasso Regularizer

The defining feature of GAMTL is the use of the Trace Lasso penalty. This regularizer acts as an adaptive bridge between the $\ell_1$ norm and the $\ell_2$ norm:

  • Uncorrelated Tasks: When tasks are independent, Trace Lasso behaves like the $\ell_1$ norm, promoting sparsity and selecting only the most relevant predictors.
  • Correlated Tasks: When tasks are highly related, it behaves like the $\ell_2$ norm, grouping them together to share information across the entire cluster.

This adaptivity allows the model to automatically discover task structures without requiring the user to define the number of clusters ($K$) beforehand. Furthermore, by allowing the correlation matrix to contain negative values, GAMTL can leverage negative correlations—where the success of one task predicts the failure of another—to improve learning.

Experimental Evaluation

The authors evaluated GAMTL against state-of-the-art baselines, including Adaptive Multi-Task Learning (AMTL) and Multi-Task Relationship Learning (MTRL), across synthetic and real-world datasets.

Structure Discovery and Clustering

In synthetic experiments designed to test group recovery, GAMTL demonstrated a superior ability to identify latent task structures. As shown in the comparison below, GAMTL successfully recovers the block-diagonal structure of task correlations, whereas baseline methods like AMTL produce sparse, disconnected results that fail to capture the group dynamics.

Comparison of learned task correlation structures between GAMTL and AMTL. GAMTL (a, c) recovers clear block-diagonal group structures, while AMTL (b, d) fails to capture the underlying task relationships.

Performance on Real-World Benchmarks

  • Multi-Domain Sentiment (MDS): On Amazon product reviews, GAMTL achieved a prediction error of 12.82%, a significant improvement over the Single Task Learning (STL) baseline of 28.77%.
  • CIFAR-10: In image classification tasks, the model successfully grouped visually similar categories (e.g., different types of vehicles vs. different animals), leading to higher accuracy than competing MTL methods.
  • School Dataset: Even in scenarios where task grouping is weak, GAMTL remained competitive with specialized low-rank methods, demonstrating its robustness.

Conclusion

GAMTL provides a robust solution for the selective sharing problem in Multi-Task Learning. By utilizing the Trace Lasso, it eliminates the need for manual cluster specification and effectively manages the trade-off between sparsity and grouping. While the optimization involves computationally intensive singular value decompositions (SVD), the model's ability to automatically discover complex task relationships marks a significant advancement in adaptive learning frameworks.

Summary updated

Abstract

In multi-task learning (MTL), tasks are learned jointly so that information among related tasks is shared and utilized to help improve generalization for each individual task. A major challenge in MTL is how to selectively choose what to share among tasks. Ideally, only related tasks should share information with each other. In this paper, we propose a new MTL method that can adaptively group correlated tasks into clusters and share information among the correlated tasks only. Our method is based on the assumption that each task parameter is a linear combination of other tasks' and the coefficients of the linear combination are active only if there is relatedness between the two tasks. Through introducing trace Lasso penalty on these coefficients, our method is able to adaptively select the subset of coefficients with respect to the tasks that are correlated to the task. Our model frees the process of determining task clustering structure as used in the literature. Efficient optimization methods based on alternating direction method of multipliers (ADMM) is developed to solve the problem. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of our method in terms of clustering related tasks and generalization performance.
BibTeX
@inproceedings{liu2017adaptive,
  author = {Sulin Liu and Sinno Jialin Pan},
  title = {{Adaptive Group Sparse Multi-task Learning via Trace Lasso}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/328},
}
Figure