Generalized Dictionary for Multitask Learning with Boosting

Boyu Wang, Joelle PineauIJCAI 2016 · OpenReview

Abstract While multitask learning has been extensively studied, most existing methods rely on linear models, which may fail in dealing with more general nonlinear problems. In this paper, we present a new approach that combines dictionary learning with gradient boosting to achieve multitask learning with general nonlinear basis functions. Specifically, for each task we learn a sparse representation in a nonlinear dictionary that is shared across the set of tasks. Each atom of the dictionary is a nonlinear feature mapping of the original input space, learned in function space by gradient boosting. The resulting model is a hierarchical ensemble where the top layer of the hierarchy is the task-specific sparse coefficients and the bottom layer is the boosted models common to all tasks. The proposed method takes the advantages of both dictionary learning and boosting for multitask learning: knowledge across tasks can be shared via the dictionary, and flexibility and generalization performance are guaranteed by boosting. Experimental results on both synthetic and benchmark real-world datasets confirm the effectiveness of the proposed approach.

Introduction

Multitask learning (MTL) aims to improve a model's performance on a specific task by leveraging information from other related tasks. However, traditional MTL methods often struggle with two major limitations: a reliance on linear models that fail to capture complex data relationships, and a rigid assumption that all tasks are related, which ignores potential outliers or sub-groups. While kernel methods offer a nonlinear alternative, they suffer from high computational complexity ($O(N^3)$) and tuning difficulties.

To address these challenges, we introduce Generalized Dictionary for Multitask Learning with Boosting (GDMTLB). This framework combines the flexibility of gradient boosting with the efficiency of dictionary learning, allowing for the discovery of sophisticated task structures and nonlinear relationships.

The GDMTLB Framework

The core innovation of GDMTLB is the replacement of traditional linear dictionary atoms with nonlinear basis functions learned in function space. The model is structured as a hierarchical ensemble:

  1. Top Layer: Task-specific sparse coefficients that determine which "atoms" in the dictionary a specific task should utilize.
  2. Bottom Layer: The dictionary atoms themselves, which are nonlinear models learned via gradient boosting.

The optimization process alternates between Sparse Coding (fixing the dictionary to find optimal task coefficients) and Dictionary Learning (using functional gradient descent to update the nonlinear atoms). This approach allows for selective knowledge sharing; because coefficients are sparse, the model naturally identifies clusters of tasks that share similar boosted atoms.

Visualization of task grouping and coefficient sparsity in the synthetic dataset.

GDMTLB acts as a versatile wrapper, capable of using various weak learners (e.g., decision trees) as base learners. Theoretical analysis shows that the model inherits the margin-maximizing benefits of AdaBoost and the dimensionality reduction benefits of dictionary learning, particularly when the number of atoms is smaller than the number of tasks.

Experimental Evaluation

We evaluated GDMTLB across synthetic and real-world benchmarks, including the London School data (regression), Landmine detection (classification), and BCI Competition EEG data.

Synthetic Data and Task Grouping

On a 2D synthetic dataset designed with nonlinearities and two distinct task groups, GDMTLB significantly outperformed competitors. It achieved an RMSE of 3.31, compared to 4.42 for the next best nonlinear method (MultiBoost) and ~5.0 for traditional linear MTL methods.

Comparison of the synthetic data in the original input space (a-c) versus the learned dictionary space (d-f).

As shown in the visualizations, the model successfully transforms the complex, nonlinear input space into a structured dictionary space where task relationships are more apparent.

Real-World Performance

In the BCI (Brain-Computer Interface) task, GDMTLB achieved a mean accuracy of 76.31%, a notable improvement over Single Task Learning (74.07%) and standard Dictionary Multitask Learning (74.92%). Despite the complexity of the alternating optimization, the algorithm typically converges within ten iterations, demonstrating practical efficiency.

Discussion and Future Work

GDMTLB provides a robust framework for nonlinear multitask learning, though it exhibits some sensitivity to dictionary initialization. While more computationally intensive than linear models, it remains significantly more scalable than kernel-based approaches. Future research directions include integrating Deep Neural Networks as base learners within the boosting framework to further enhance the expressiveness of multitask constraints.

Summary updated

Abstract

While multitask learning has been extensively studied, most existing methods rely on linear models (e.g. linear regression, logistic regression), which may fail in dealing with more general (nonlinear) problems. In this paper, we present a new approach that combines dictionary learning with gradient boosting to achieve multitask learning with general (nonlinear) basis functions. Specifically, for each task we learn a sparse representation in a nonlinear dictionary that is shared across the set of tasks. Each atom of the dictionary is a nonlinear feature mapping of the original input space, learned in function space by gradient boosting. The resulting model is a hierarchical ensemble where the top layer of the hierarchy is the task-specific sparse coefficients and the bottom layer is the boosted models common to all tasks. The proposed method takes the advantages of both dictionary learning and boosting for multitask learning: knowledge across tasks can be shared via the dictionary, and flexibility and generalization performance are guaranteed by boosting. More important, this general framework can be used to adapt any learning algorithm to (nonlinear) multitask learning. Experimental results on both synthetic and benchmark real-world datasets confirm the effectiveness of the proposed approach for multitask learning.
BibTeX
@inproceedings{wang2016generalized,
  author = {Boyu Wang and Joelle Pineau},
  title = {{Generalized Dictionary for Multitask Learning with Boosting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=ByWzaVG_ZB},
}
Figure