A Probabilistic Model for Dirty Multi-task Feature Selection

Daniel Hernández-Lobato, José Miguel Hernández-Lobato, Zoubin GhahramaniICML 2015 · OpenReview

In the world of machine learning, Multi-Task Learning (MTL) is often treated as a "rising tide lifts all boats" scenario. The assumption is simple: if you have several related tasks, learning them together should help you identify a shared set of relevant features. But what happens when one of those tasks is a "black sheep" that doesn't follow the rules? Or when a feature is relevant to almost everyone, except for one specific task?

This paper introduces a probabilistic framework to handle these "dirty" data scenarios where the standard assumptions of multi-task feature selection fall apart.

The Problem with "Clean" Assumptions

Traditional multi-task feature selection methods usually rely on the "joint sparsity" hypothesis: the idea that if a feature is useful for Task A, it’s probably useful for Task B, C, and D as well. In practice, this is often too restrictive.

Real-world datasets are "dirty." You might encounter:

  1. Outlier Tasks: A task that has a completely different set of relevant features compared to the others.
  2. Outlier Features: A feature that is generally irrelevant across tasks but becomes highly relevant for one or two specific cases.
Visualization of coefficient matrices showing shared features alongside outlier tasks and features.

Standard methods (like Group LASSO) try to force these outliers into the shared mold, which leads to poor predictive performance and "smears" the feature selection process. To solve this, the authors propose Dirty Multi-task Feature Selection (DMFS), a probabilistic model that uses binary latent variables to automatically identify outlier tasks and outlier features.

Methods: Building a "Dirty" Prior

The authors build a hierarchical Bayesian model centered around a robust prior distribution. The core innovation is the use of five types of binary latent variables to describe the relationship between features and tasks:

  • $z_i$: Is feature $i$ an outlier?
  • $\omega_k$: Is task $k$ an outlier?
  • $\gamma_i$: Is the (non-outlier) feature $i$ relevant for all non-outlier tasks?
  • $\tau_i^{(k)}$: If task $k$ is an outlier, is feature $i$ relevant for it?
  • $\eta_i^{(k)}$: If feature $i$ is an outlier, is it relevant for task $k$?

By combining these, the model can decide whether a coefficient $w_i^{(k)}$ should be zero or drawn from a "spike-and-slab" prior. Specifically, they use the Strawderman-Berger distribution, which has heavy tails to avoid over-shrinking large coefficients.

Probability density plots comparing the proposed discrete mixture prior against standard Gaussian and Laplace priors, highlighting its heavy tails.

Inference via Expectation Propagation (EP)

Because exact Bayesian inference is impossible with this many discrete variables, the authors use Expectation Propagation. They approximate the complex posterior with a simpler distribution (Gaussian for weights, Bernoulli for latent variables). A key technical contribution is showing that the convolution of their chosen prior with a Gaussian likelihood has a closed-form solution, making the algorithm computationally efficient.

Key Insights and Results

The model was evaluated on synthetic data, gene regulatory networks, natural image denoising, and anticancer drug sensitivity datasets.

  • Superior Accuracy: On synthetic data specifically designed with outlier tasks and features, DMFS achieved a Root Mean Squared Error (RMSE) of 0.73, significantly beating standard Multi-Task Feature Selection (0.81) and Single-Task Learning (0.78).
  • Biological Discovery: In the gene network experiments, DMFS achieved the highest AUROC (0.84), successfully identifying "hub nodes" (genes that regulate many others) while allowing for task-specific variations.
  • Identifying Outliers: In the drug sensitivity study, the model flagged specific drugs (like Topotecan and Erlotinib) as outlier tasks, suggesting their sensitivity profiles are driven by different genetic markers than the rest of the drug suite.
  • The Power of Heavy Tails: By using the Strawderman-Berger distribution instead of a Laplace prior (standard LASSO), the model avoids the "shrinkage bias" where relevant coefficients are pushed too close to zero.
  • Automatic Hyperparameter Tuning: Because it's a fully Bayesian model with hyper-priors, it learns the sparsity levels and noise variances from the data itself, rather than requiring expensive cross-validation.

Limitations and Future Work

While EP is efficient, the computational cost still scales as $O(K N_k^2 d)$. For extremely high-dimensional genomic data with thousands of tasks, further optimizations or parallelization would be needed. Additionally, while DMFS was better at identifying outliers in the drug sensitivity experiment, some specialized non-probabilistic "dirty" models performed slightly better on raw RMSE, suggesting that the model's noise assumptions might need further refinement for extremely noisy biological data. Future directions could include extending this to non-linear models or incorporating more complex task correlations beyond simple "inlier/outlier" binary logic.

Summary updated

Abstract

Multi-task feature selection methods often make the hypothesis that learning tasks share relevant and irrelevant features. However, this hypothesis may be too restrictive in practice. For example, ...
BibTeX
@inproceedings{hernandezlobato2015probabilistic,
  author = {Daniel Hernández-Lobato and José Miguel Hernández-Lobato and Zoubin Ghahramani},
  title = {{A Probabilistic Model for Dirty Multi-task Feature Selection}},
  booktitle = {International Conference on Machine Learning},
  year = {2015},
  url = {https://openreview.net/forum?id=rJb_LsWOWB},
}
Figure