Group LASSO with Asymmetric Structure Estimation for Multi-Task Learning

Saullo H. G. Oliveira, André R. Gonçalves, Fernando J. Von ZubenIJCAI 2019 · DOI

TL;DR

This paper introduces Group Asymmetric Multi-Task Learning (GAMTL), a novel framework that allows tasks to share information in a more flexible and realistic way. Unlike previous methods that assumed symmetric relationships between tasks, GAMTL learns asymmetric, group-specific transference structures, leading to improved interpretability and reduced negative transfer.

What Problem Does This Paper Address?

Multi-Task Learning (MTL) aims to improve learning by leveraging information from related tasks. A common approach is to use regularization techniques like Group LASSO, which encourages sparsity within groups of features. However, standard Group LASSO implicitly assumes all tasks are related in the same way, which can lead to "negative transfer" – where learning from one task actually harms performance on another. While some methods try to model task relationships, they often assume these relationships are symmetric (if task A influences task B, then B influences A equally) and uniform across all features. This paper addresses the need for a more nuanced approach that can capture asymmetric and group-specific relationships between tasks, allowing for more accurate and interpretable knowledge sharing.

Methods

The core contribution is the Group Asymmetric Multi-Task Learning (GAMTL) formulation. It extends Group LASSO by introducing an asymmetric transference structure that is learned per group of features.

Here's a breakdown of the key methodological aspects:

  • Group-Specific Asymmetric Transference: Instead of a single symmetric matrix for all tasks, GAMTL learns a separate asymmetric matrix ($B^g$) for each group of features ($g$). This matrix dictates how tasks influence each other within that specific group.
  • Optimization: The optimization problem is non-convex but is solved efficiently using an alternating optimization procedure. This involves iteratively optimizing for task parameters ($W$) and transference matrices ($B^g$).
  • Solving for Task Parameters ($W$): This sub-problem is solved using the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), a standard method for solving problems with smooth and non-smooth convex components.
  • Solving for Transference Matrices ($B^g$): This sub-problem is addressed using the Alternating Direction Method of Multipliers (ADMM), a powerful technique for solving optimization problems with coupled constraints, particularly useful here for handling the non-negativity constraint on the transference coefficients.
  • Flexibility: The framework allows for overlapping groups of features and can optionally enforce non-negativity on the transference coefficients, meaning tasks can only positively influence each other.

Datasets

The paper uses two types of datasets:

  • Artificial Dataset: A synthetic dataset was generated to illustrate the model's components and validate its ability to recover known transference structures. It features 8 regression tasks with 50 attributes partitioned into two groups.
  • Real Dataset: The ADNI dataset was used, which is a well-known dataset for Alzheimer's Disease research. This dataset involves predicting 5 cognitive scores from structural MRI data, with features organized into 116 brain regions of interest (ROIs), some of which are grouped.

Key Insights and Ideas

  • Asymmetry is Crucial: The assumption of symmetric task relationships is often too restrictive. Real-world scenarios frequently involve asymmetric influences, where one task might be a stronger predictor or source of information for another.
  • Group-Specific Relationships: Different groups of features might have entirely different underlying relationships between tasks. For example, one brain region might be highly predictive of one cognitive score but irrelevant for another. GAMTL captures this by learning separate transference matrices for each feature group.
  • Interpretability through Structure: The learned asymmetric transference matrices ($B^g$) provide a direct and interpretable view of how tasks influence each other within specific feature groups. This is a significant advantage over methods that learn latent structures without clear meaning.
  • Reducing Negative Transfer: By allowing for more flexible and accurate modeling of task relationships, GAMTL is better equipped to avoid negative transfer, leading to improved overall performance.

Key Results

  • Artificial Data: GAMTL successfully recovered the known asymmetric transference structures on the synthetic data, demonstrating its ability to learn these relationships.
  • ADNI Dataset:
    • GAMTL achieved the best overall performance in terms of Normalized Mean Squared Error (NMSE) compared to several state-of-the-art methods, including LASSO, Group LASSO, MT-SGL, MTFL, MTRL, and AMTL.
  • While LASSO performed slightly better on two specific tasks (RAVLT TOTAL and T30), GAMTL outperformed other methods on the majority of tasks.
  • The learned transference structures on the ADNI dataset provided meaningful insights into how different brain regions (ROIs) relate to cognitive scores, aligning with existing AD research.

Limitations and Future Work

  • Computational Complexity: The paper notes that the complexity of an iteration of GAMTL is driven by FISTA and ADMM executions, resulting in an overall time complexity of $O(T^3Gn + T^2Gn^2)$, which can be substantial for a large number of tasks ($T$) and groups ($G$).
  • Future Work: The authors suggest exploring new applications, investigating other restrictions on the relationship matrices (beyond non-negativity), and developing alternative visualization methods for the estimated structures. They also mention the possibility of using GAMTLnr (GAMTL without the non-negativity constraint) when negative relationships are meaningful.

Summary updated

BibTeX
@inproceedings{oliveira2019group,
  author = {Saullo H. G. Oliveira and André R. Gonçalves and Fernando J. Von Zuben},
  title = {{Group LASSO with Asymmetric Structure Estimation for Multi-Task Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/444},
}
Figure