Kernelized Bayesian Transfer Learning

Mehmet Gönen, Adam A. MargolinAAAI 2014 · DOI

Introduction

Transfer learning aims to improve the performance of a target task by leveraging knowledge from a related source task. This is particularly crucial when labeled data for the target task is scarce. However, a significant challenge arises in heterogeneous domains, where source and target data possess different feature representations or distributions. Existing methods often rely on disjoint two-step processes—dimensionality reduction followed by classification—or lack a rigorous probabilistic foundation.

This paper introduces Kernelized Bayesian Transfer Learning (KBTL), a unified, fully Bayesian framework designed for heterogeneous transfer learning. KBTL jointly learns task-specific projections into a shared latent subspace and a common classifier, enabling effective knowledge transfer even across disparate feature spaces.

Methodology

The KBTL framework integrates two core components into a single discriminative model:

  1. Kernel-Based Dimensionality Reduction: To handle heterogeneous features, the model assigns a unique kernel matrix and projection matrix to each task. This allows data from different domains to be mapped into a shared, lower-dimensional "hidden" subspace. The use of kernels facilitates the modeling of non-linear relationships and high-dimensional inputs.
  2. Coupled Classification: Within the shared subspace, the model learns a single linear classifier (shared weights and bias) across all tasks. This coupling forces the dimensionality reduction step to identify a subspace where the classification logic remains consistent across domains.
The KBTL architecture: task-specific kernel projections into a shared subspace followed by a common classifier.

To ensure scalability, the authors derive an efficient Variational Bayes (VB) inference procedure. Unlike computationally intensive Gibbs sampling, VB provides a fast approximation of the posterior distributions, making the model viable for large-scale, real-world applications.

Experiments and Datasets

The efficacy of KBTL was evaluated on two distinct benchmarks:

  • Object Recognition (Computer Vision): Utilizing the Office (Amazon, Webcam, DSLR) and Caltech-256 datasets, the study focused on 10 common object categories. This benchmark tests the model's ability to adapt across different imaging conditions and domains using 800-dimensional SURF features.
  • Cancer Genomics (Computational Biology): A more challenging "heterogeneous" problem involves predicting gene mutation status (e.g., TP53, EGFR) by transferring knowledge from in-vitro cancer cell lines (CCLE) to primary patient tumors (TCGA). These domains exhibit vastly different biological characteristics, requiring robust transfer mechanisms.

Key Results

KBTL demonstrates superior performance across both domains. In computer vision tasks, KBTL outperformed state-of-the-art baselines, including GFK and MMDT, in 9 out of 12 domain adaptation experiments.

Performance comparison of KBTL against state-of-the-art transfer learning methods on the Office+Caltech-256 dataset.

In cancer genomics, the model successfully bridged the gap between cell lines and patient data, significantly outperforming "no-transfer" baselines like Bayesian Relevance Vector Machines. Key insights from the results include:

  • Joint Optimization: Simultaneous learning of projections and classifiers ensures the shared subspace is task-relevant.
  • Bayesian Robustness: The fully Bayesian approach naturally quantifies uncertainty, providing more reliable probabilistic outputs.
  • Versatility: The framework maintains high accuracy regardless of whether linear or non-linear Gaussian kernels are employed.

Conclusion and Future Work

KBTL provides a principled approach to heterogeneous transfer learning. While highly effective, the authors identify potential improvements, such as incorporating Multiple Kernel Learning (MKL) to automatically select feature representations and implementing "task grouping" to mitigate the risk of negative transfer when tasks are overly dissimilar. Future work may also extend the model to semi-supervised settings to leverage unlabeled data.

Summary updated

BibTeX
@inproceedings{gonen2014kernelized,
  author = {Mehmet Gönen and Adam A. Margolin},
  title = {{Kernelized Bayesian Transfer Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.8948},
}
Figure