Infinite Predictor Subspace Models for Multitask Learning

Piyush Rai, Hal Daumé IiiAISTATS 2010 · OpenReview

Abstract

Given several related learning tasks, we propose a nonparametric Bayesian model that captures task relatedness by assuming that the task parameters (i.e., predictors) share a latent subspace. By utilizing the Indian Buffet Process (IBP), the model automatically determines the intrinsic dimensionality and sparsity of this shared space. We further extend this framework to an augmented model that shares the latent subspace between input data and task parameters, enabling the effective use of unlabeled data.

Introduction

Multitask Learning (MTL) aims to improve the generalization performance of multiple related tasks by sharing information across them. This is particularly beneficial when labeled data for individual tasks is scarce. A common approach in MTL is to assume that task-specific parameters lie in a low-dimensional shared subspace. However, traditional methods often require the user to pre-define the subspace dimensionality ($K$). An incorrect choice of $K$ can lead to either the loss of task-specific nuances or overfitting to noise. Furthermore, forcing unrelated tasks to share a subspace can result in "negative transfer," where learning one task degrades performance on another.

Methodology

The proposed framework addresses these challenges by employing a nonparametric Bayesian approach that avoids manual tuning of subspace complexity.

The Infinite Subspace via IBP

Instead of fixing the number of basis vectors, we use the Indian Buffet Process (IBP) to allow the model to discover the dimensionality of the task subspace from the data. The task parameters are represented as linear combinations of a set of basis tasks. The task basis $Z$ is modeled as the element-wise product of a binary matrix $B$ (sampled from the IBP) and a real-valued matrix $V$. This formulation ensures a sparse basis, effectively selecting only the most relevant shared features.

Model Variants

We present two primary models:

  1. Model-1: A basic subspace model where task parameters share a latent basis.
  2. Model-2: An augmented model that, motivated by the Representer Theorem, assumes the input data ($X$) and task parameters ($\theta$) share the same latent subspace. This allows the model to leverage unlabeled data to better define the subspace, improving supervised learning performance.
Graphical representation of Model-1 (left) and Model-2 (right).

Experiments and Results

The models were evaluated on two multi-label classification benchmarks: Yeast (14 tasks, 103 features) and Scene (6 tasks, 294 features). These datasets represent typical MTL scenarios where instances are associated with multiple related labels.

Performance Comparison

We compared Model-1 and Model-2 against independent Logistic Regression (LR), data pooling, and a state-of-the-art matrix stick-breaking baseline ("yaxue").

Performance metrics for various models on the Yeast and Scene datasets.

As shown in the results, Model-2 consistently achieves superior accuracy and F1 scores. On the Yeast dataset, Model-2 reached an accuracy of 0.5424, significantly outperforming independent LR (0.5047). Notably, simple data pooling often performed worse than independent learning, highlighting the importance of the nuanced subspace approach. Experimental analysis across varying training set sizes also demonstrated that the proposed models reach near-optimal performance much faster than independent models, highlighting their efficiency in small-data regimes.

Conclusion

The infinite predictor subspace model provides a robust framework for MTL by automatically controlling model complexity and mitigating negative transfer. The integration of unlabeled data through shared subspaces further enhances performance. Future work includes exploring faster inference methods like variational approximations to scale the model to larger datasets and extending the framework to handle non-linear task relationships through mixtures of subspaces.

Summary updated

Abstract

Given several related learning tasks, we propose a nonparametric Bayesian model that captures task relatedness by assuming that the task parameters (i.e., predictors) share a latent subspace. More ...
BibTeX
@inproceedings{rai2010infinite,
  author = {Piyush Rai and Hal Daumé III},
  title = {{Infinite Predictor Subspace Models for Multitask Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2010},
  url = {https://openreview.net/forum?id=zbOlnTmzBn},
}
Figure