Radial Basis Function Network for Multi-task Learning

Xuejun Liao, Lawrence CarinNeurIPS 2005

Overview

This paper introduces a framework for Multi-task Learning (MTL) using Radial Basis Function (RBF) networks. By allowing multiple correlated tasks to share a single hidden layer of basis functions while maintaining task-specific output weights, the model effectively transfers knowledge across tasks. The authors also provide an active learning strategy to intelligently select which data points to label, significantly improving performance when training data is scarce.

In many real-world machine learning scenarios, we face multiple related tasks rather than a single isolated one. For example, predicting student performance across different schools involves distinct but highly correlated tasks.

The primary challenge is that individual tasks often have "weak" training sets—too little data to build a robust model independently. Standard approaches either treat every task as independent (ignoring valuable correlations) or lump all data together into one giant task (ignoring task-specific nuances). This paper addresses this gap by creating a unified representation that exploits the transferability of knowledge between tasks without losing the unique characteristics of each.

Proposed Method: Multi-task RBF Network

The authors propose a novel Multi-task RBF Network architecture designed to balance shared representations with task-specific tuning.

Architecture of the Multi-task RBF Network
  • Shared Hidden Layer: All tasks share the same set of hidden nodes (radial basis functions). These nodes capture the common internal representation or "features" relevant to all tasks.
  • Task-Specific Outputs: Each task has its own output node with a unique set of weights connecting it to the shared hidden layer. This allows the model to specialize for each task.
  • Sequential Learning Algorithm: Instead of training everything at once, the authors develop an algorithm to select basis functions one by one from a candidate pool to minimize the total squared error across all tasks.
  • Active Learning: The paper derives a mathematical bound for the generalization error. Using this, they develop a method to actively select the most informative unsupervised data points to be labeled, ensuring the model learns effectively with fewer supervised examples.

Evaluation and Datasets

The primary evaluation uses the Inner London Education Authority (ILEA) school dataset, which consists of examination records for 15,362 students across 139 secondary schools. The goal is to predict exam scores based on 9 variables (expanded into 27 binary input variables). Each of the 139 schools is treated as a separate "task."

Key Insights

  • The "Sweet Spot" of Sharing: Sharing the basis functions (the hidden layer) while keeping weights independent is superior to both total independence and total pooling. It captures the underlying structure of the data distribution while allowing for task-specific offsets.
  • Mutual Enhancement: Multi-task learning is most beneficial when individual training sets are small. By learning simultaneously, a "weak" task can borrow structural information from a "stronger" task.
  • Active Selection via Novelty: The active learning component identifies "novel" data points—those significantly dissimilar to the current training set—and prioritizes them for labeling, preventing wasted effort on redundant data.

Key Results

The authors compared three models: the Multi-task RBF, Independent RBFs (one per school), and "One RBF" (all schools pooled into one task).

Performance comparison under varying training data sizes
  • Superior Accuracy: The Multi-task RBF achieved a squared error of 109.89, significantly lower than Independent RBFs (136.41) and the pooled "One RBF" (149.48).
  • Robustness to Data Scarcity: As shown in the active learning tests, the Multi-task RBF maintained high performance even as the number of training points dropped toward 5,000. In contrast, the independent models' performance degraded rapidly with less data.
  • Efficiency: By sharing the hidden layer, the resulting network is more compact than a collection of independent networks, leading to better generalization.

Limitations and Future Work

  • Computational Complexity: While the sequential selection of basis functions is efficient, the matrix inversions required for $K$ tasks could become a bottleneck if the number of tasks or basis functions grows extremely large.
  • RBF Parameter Tuning: The paper focuses on Gaussian RBFs. While they tune the width ($\sigma$), performance might vary with different types of basis functions or more complex kernel choices.
  • Task Correlation Assumption: The method assumes tasks are correlated. If tasks are entirely independent, the multi-task approach offers no benefit over independent learning. Future work could explore automatically detecting the degree of correlation to weight task influence dynamically.

Summary updated

BibTeX
@inproceedings{liao2005radial,
  author = {Xuejun Liao and Lawrence Carin},
  title = {{Radial Basis Function Network for Multi-task Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2005},
}
Figure