Non-Linear Domain Adaptation with Boosting

Carlos J. Becker, C. Mario Christoudias, Pascal Fua

Abstract

This paper introduces a novel boosting-based algorithm for non-linear domain adaptation. By leveraging a "boosting-trick," the method learns to map data from different domains into a shared feature space where a single, common decision boundary can be applied. This approach significantly improves classification performance in scenarios where labeled data is scarce in the target domain.

Introduction: The Domain Shift Problem

A fundamental assumption in machine learning is that training (source) and test (target) data are drawn from the same distribution. In fields like bio-medical imaging, this assumption is frequently violated. For example, 3D Electron Microscopy (EM) scans of different brain regions (like the Striatum versus the Hippocampus) may look substantially different due to varying experimental conditions, even if the underlying task—such as segmenting mitochondria—remains identical.

Labeling 3D data is incredibly time-consuming and expensive. When researchers acquire a new dataset, they often have very few labeled examples. Traditional classifiers trained on a previous dataset generalize poorly to the new one. The core challenge is finding a way to transfer knowledge from the label-rich source domain to the label-poor target domain, accounting for the non-linear transformations that occur between different data acquisitions.

Methodology

The authors propose a multi-task learning approach specifically tailored for domain adaptation.

  1. Shared Feature Space: Unlike standard multi-task learning, which often learns different decision boundaries for different tasks, this method assumes the underlying task is identical. It seeks to learn a non-linear mapping for each task that projects observations into a common feature space where a single linear decision boundary can separate the classes.
  2. The "Boosting-Trick": Modeling non-linear transformations usually requires kernel methods, which are computationally expensive and require picking a specific kernel function a-priori. Instead, the authors use Gradient Boosting. They represent the non-linear mapping as an ensemble of weak learners (regression trees or decision stumps).
  3. Joint Optimization: The algorithm jointly optimizes the task-specific mappings and the shared decision boundary. At each iteration, it finds a weak learner and a set of task-specific thresholds that minimize the exponential loss across all domains simultaneously.
  4. Efficiency: By using decision trees that operate on single dimensions, the method remains highly efficient even with the extremely high-dimensional feature vectors (100,000+ dimensions) common in bio-medical imaging.

Datasets and Evaluation

The authors evaluated their approach on two challenging bio-medical problems to test its robustness across modalities and dimensions:

  • Path Classification: Detecting 3D curvilinear structures (neural axons) using a source domain of 2D aerial road images. This tests the model's ability to adapt across entirely different dimensions and visual modalities.
Target domain for path classification: 3D curvilinear structures (neural axons).
  • Mitochondria Segmentation: Segmenting organelles in 3D EM stacks. The source domain was a stack from the Striatum (with 39 labeled mitochondria), and the target domain was a stack from the Hippocampus (simulating a scenario with very few labels).
Source domain example: Electron Microscopy (EM) scan for mitochondria segmentation.

Key Insights

  • The Power of a Single Boundary: The core insight is that for domain adaptation, task-specific boundaries are often a sign of overfitting to domain noise. By forcing a single shared boundary in a learned latent space, the model is better regularized and can learn effectively from fewer target samples.
  • Parameter-Free Non-Linearity: Using boosting as a "trick" to handle non-linearity is a clever alternative to kernels. It avoids the need to guess the right kernel function and scales much better to large datasets.
  • Handling High Dimensionality: The method is robust enough to handle feature vectors with over 100,000 dimensions, a regime where traditional kernel-based latent variable models often become computationally infeasible.

Results

  • Rapid Adaptation: In the Path Classification task, the proposed method reached the performance of a fully supervised model (trained on the entire target dataset) using as few as 70 training samples.
  • Outperforming Baselines: On the Mitochondria task, the method significantly outperformed standard techniques like Mean-Variance Normalization (MVN) and Histogram Matching (HM), as well as other state-of-the-art multi-task learning algorithms.
  • Computational Speed: For the mitochondria dataset, the proposed approach took less than one hour to run, whereas a competing multi-task boosting method took over seven hours due to the complexity of its cross-validation requirements.

Limitations and Future Work

While the paper demonstrates strong results, a few areas remain for exploration:

  • Transformation Model: The current model focuses on task-specific shifts. While the ensemble of trees can model complex non-linearities, more explicit modeling of other types of domain shifts (like rotations or scaling) might further improve performance.
  • Weak Learner Selection: Performance is tied to the choice of weak learners. While trees are generally robust, very high-dimensional data might favor simpler stumps to reduce training time.
  • Unsupervised Adaptation: The current method requires at least a few labels in the target domain. A promising future direction would be extending this boosting framework to purely unsupervised domain adaptation, where no target labels are available at all.

Summary updated

Abstract

A common assumption in machine vision is that the training and test samples are drawn from the same distribution. However, there are many problems when this assumption is grossly violated, as in bio-medical applications where differ-ent acquisitions can generate drastic variations in the appearance of the data due to changing experimental conditions. This problem is accentuated with 3D data, for which annotation is very time-consuming, limiting the amount of data that can be labeled in new acquisitions for training. In this paper we present a multi-task learning algorithm for domain adaptation based on boosting. Unlike previous approaches that learn task-specific decision boundaries, our method learns a sin-gle decision boundary in a shared feature space, common to all tasks. We use the boosting-trick to learn a non-linear mapping of the observations in each task, with no need for specific a-priori knowledge of its global analytical form. This yields a more parameter-free domain adaptation approach that successfully lever-ages learning on new tasks where labeled data is scarce. We evaluate our approach on two challenging bio-medical datasets and achieve a significant improvement over the state of the art. 1
BibTeX
@inproceedings{becker2013non,
  author = {Carlos J. Becker and C. Mario Christoudias and Pascal Fua},
  title = {{Non-Linear Domain Adaptation with Boosting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
}
Figure