Abstract
Given several related learning tasks, we propose a nonparametric Bayesian model that captures task relatedness by assuming that the task parameters (i.e., predictors) share a latent subspace. By utilizing the Indian Buffet Process (IBP), the model automatically determines the intrinsic dimensionality and sparsity of this shared space. We further extend this framework to an augmented model that shares the latent subspace between input data and task parameters, enabling the effective use of unlabeled data.
Introduction
Multitask Learning (MTL) aims to improve the generalization performance of multiple related tasks by sharing information across them. This is particularly beneficial when labeled data for individual tasks is scarce. A common approach in MTL is to assume that task-specific parameters lie in a low-dimensional shared subspace. However, traditional methods often require the user to pre-define the subspace dimensionality ($K$). An incorrect choice of $K$ can lead to either the loss of task-specific nuances or overfitting to noise. Furthermore, forcing unrelated tasks to share a subspace can result in "negative transfer," where learning one task degrades performance on another.
Methodology
The proposed framework addresses these challenges by employing a nonparametric Bayesian approach that avoids manual tuning of subspace complexity.
The Infinite Subspace via IBP
Instead of fixing the number of basis vectors, we use the Indian Buffet Process (IBP) to allow the model to discover the dimensionality of the task subspace from the data. The task parameters are represented as linear combinations of a set of basis tasks. The task basis $Z$ is modeled as the element-wise product of a binary matrix $B$ (sampled from the IBP) and a real-valued matrix $V$. This formulation ensures a sparse basis, effectively selecting only the most relevant shared features.
Model Variants
We present two primary models:
- Model-1: A basic subspace model where task parameters share a latent basis.
- Model-2: An augmented model that, motivated by the Representer Theorem, assumes the input data ($X$) and task parameters ($\theta$) share the same latent subspace. This allows the model to leverage unlabeled data to better define the subspace, improving supervised learning performance.
Experiments and Results
The models were evaluated on two multi-label classification benchmarks: Yeast (14 tasks, 103 features) and Scene (6 tasks, 294 features). These datasets represent typical MTL scenarios where instances are associated with multiple related labels.
Performance Comparison
We compared Model-1 and Model-2 against independent Logistic Regression (LR), data pooling, and a state-of-the-art matrix stick-breaking baseline ("yaxue").
As shown in the results, Model-2 consistently achieves superior accuracy and F1 scores. On the Yeast dataset, Model-2 reached an accuracy of 0.5424, significantly outperforming independent LR (0.5047). Notably, simple data pooling often performed worse than independent learning, highlighting the importance of the nuanced subspace approach. Experimental analysis across varying training set sizes also demonstrated that the proposed models reach near-optimal performance much faster than independent models, highlighting their efficiency in small-data regimes.
Conclusion
The infinite predictor subspace model provides a robust framework for MTL by automatically controlling model complexity and mitigating negative transfer. The integration of unlabeled data through shared subspaces further enhances performance. Future work includes exploring faster inference methods like variational approximations to scale the model to larger datasets and extending the framework to handle non-linear task relationships through mixtures of subspaces.
Summary updated