Learning from Relevant Tasks Only

Samuel Kaski, Jaakko Peltonen

In machine learning, the "small data" problem is a persistent challenge. Whether classifying rare proteins or predicting specific user preferences, researchers often have limited labeled examples for a task-of-interest but access to vast "background data" from related tasks. Standard Multi-Task Learning (MTL) typically treats all tasks as equal partners, which can lead to "negative transfer" if auxiliary tasks contain irrelevant noise.

Samuel Kaski and Jaakko Peltonen address this in their work on Relevant Subtask Learning, proposing a surgical approach that selectively borrows only useful information.

The Problem: When More Data Isn't Better

Most classifiers assume training data is representative of the test distribution. In practice, we often have a mix of representative and partially representative data. Existing MTL methods often use shared priors or task clustering, but these are usually symmetric—treating every task as equally important. If the task-of-interest is unique, or if auxiliary tasks contain distracting information, standard MTL can degrade performance. The goal is to extract useful subtasks from background data without allowing irrelevant components to distort the target model.

Methodology: The Relevant Subtask Model (RSM)

The authors introduce the Relevant Subtask Model (RSM), an asymmetric approach to MTL. The core innovation lies in how supplementary tasks are modeled:

  1. Task-of-Interest ($U$): Modeled using standard logistic regression.
  2. Supplementary Tasks ($S$): Each is modeled as a mixture of a Relevant Part (sharing parameters with the task-of-interest) and an Irrelevant Part (using unique parameters to "soak up" non-matching data).
  3. Mixture Weight ($\pi_S$): Learns the proportion of task $S$ relevant to task $U$.

By providing a "trash can" for irrelevant data through task-specific parameters, the shared parameters are free to focus exclusively on the signal overlapping with the target task.

Experimental Results

The paper evaluates RSM using synthetic "continuum" domains and the Reuters-21578 news collection.

Figure 1: Performance of RSM vs. baselines across synthetic domains with varying task clustering.

In synthetic tests (Figure 1), as tasks became less clustered—a scenario where traditional MTL (like TCM) struggles—RSM maintained high performance, staying close to the theoretical upper bound. It consistently outperformed "Single-task learning" (which overfits) and "All together" pooling (which suffers from noise).

Figure 5: Performance in scenarios without a common subtask, demonstrating RSM's robustness.

Even when auxiliary tasks did not perfectly share a subtask (Figure 5), RSM proved robust. On real-world Reuters news data, RSM effectively identified overlapping interests among users to build a better classifier for the target user, even with very few labeled examples.

Key Insights

  • Asymmetry as a Feature: RSM prioritizes the target task over auxiliary task performance.
  • Explaining Away Noise: Modeling the irrelevant component prevents it from "poisoning" the shared parameters.
  • Robustness: The model handles failures in task-sharing assumptions better than naive pooling or symmetric clustering.

Conclusion and Future Directions

RSM provides a robust framework for data-starved tasks by surgically extracting relevant information from auxiliary sources. While demonstrated with logistic regression, the authors suggest extending this to more complex models or a full Bayesian treatment to further enhance robustness.

Summary updated

Abstract

We introduce a problem called relevant subtask learning, a variant of multi-task learning. The goal is to build a classifier for a task-of-interest having too little data. We also have data for other tasks but only some are relevant, meaning they contain samples classified in the same way as in the task-of-interest. The problem is how to utilize this “background data” to improve the classifier in the task-of-interest. We show how to solve the problem for logistic regression classifiers, and show that the solution works better than a comparable multi-task learning model. The key is to assume that data of all tasks are mixtures of relevant and irrelevant samples, and model the irrelevant part with a sufficiently flexible model such that it does not distort the model of relevant data.
BibTeX
@inproceedings{kaski2007learning,
  author = {Samuel Kaski and Jaakko Peltonen},
  title = {{Learning from Relevant Tasks Only}},
  booktitle = {ECML},
  year = {2007},
  doi = {10.1007/978-3-540-74958-5\_59},
}
Figure