Supervised Heterogeneous Domain Adaptation via Random Forests

Sanatan Sukhija, Narayanan Chatapuram Krishnan, Gurkanwal SinghIJCAI 2016

Introduction

In traditional machine learning, models typically assume that training (source) and testing (target) datasets share the same feature space. However, real-world applications often encounter Heterogeneous Domain Adaptation (HDA), where features differ in dimension and semantic meaning. For instance, a 'Smart Home A' might utilize 20 motion sensors, while 'Smart Home B' uses only 10 sensors in different locations. This paper introduces SHDA-RF, a novel algorithm designed to bridge these disparate feature spaces by identifying shared label distributions as 'pivots.' Unlike existing solutions that rely on complex synthetic error-correcting codes, SHDA-RF finds a 'common language' through the natural partitioning of data.

Methodology

The core innovation of SHDA-RF is using label distributions at the leaf nodes of Random Forests as a bridge between domains. The process involves four key steps:

  1. Estimating Pivots: Random Forests are trained on both the source and target domains. Leaf nodes with similar class label distributions across the source and target forests are identified as 'pivots.' These pivots represent semantic commonalities that transcend specific feature sets.
  2. Feature Relationship Matrices ($W_S$ and $W_T$): The algorithm quantifies the contribution of each feature to specific label distributions along the tree paths. Features used higher in the tree hierarchy are given more weight, reflecting their greater discriminative power.
  3. Learning the Mapping ($P_S$): Using LASSO (Least Absolute Shrinkage and Selection Operator), the model learns a sparse linear transformation. This mapping represents each target feature as a combination of a few relevant source features, mirroring physical realities where a specific sensor in one environment likely corresponds to only a handful of sensors in another.
  4. Final Prediction: Source data is projected into the target feature space using the learned mapping, and a final model is trained on the combined dataset to predict labels for the target domain.
The SHDA-RF framework uses label distributions at leaf nodes as pivots to learn feature relationship matrices ($W_S$, $W_T$) and bridge heterogeneous domains.

Key Insights

  • Labels as the Universal Bridge: While features change across domains, the underlying tasks (the labels) remain constant. By focusing on the label distributions that features produce, the algorithm finds semantic commonality without manual feature engineering.
  • Random Forests as Natural Clusterers: Instead of using complex mathematical transformations to find latent spaces, SHDA-RF leverages the natural partitioning of data that occurs within a Random Forest.
  • The Power of Sparsity: The use of LASSO reflects the assumption that a feature in the target domain is likely related to only a small subset of features in the source domain, which improves model interpretability and reduces noise.

Datasets and Evaluation

The authors validated SHDA-RF across three diverse types of data:

  • CASAS (Horizon House): A smart home activity recognition dataset where different houses have different sensor layouts and counts.
  • 20 Newsgroups: A high-dimensional text collection where the vocabulary is split to force the model to transfer knowledge between different sets of words.
  • Statlog (Landsat Satellite): An image dataset with real-valued features, used to test the algorithm's performance on dense, non-sparse data.

Key Results

SHDA-RF consistently outperformed baseline classifiers (standard Random Forests and SVMs) and state-of-the-art transfer methods like SHFR-ECOC and HeMAP.

  • Accuracy Gains: In smart home environments, SHDA-RF improved accuracy by approximately 2-3% over the next best transfer method (p-value < 0.05). In text classification tasks, the performance gain was even more dramatic, reducing mean error from 46.61% to 40.06% for specific tasks.
  • Data Efficiency: The model showed significant resilience when target labeled data was extremely limited (e.g., only 10 samples per class), which is the most common bottleneck in real-world domain adaptation.
Mean error comparison between Baseline Random Forest (BRF) and SHDA-RF for the hh102 to hh113 transfer task, demonstrating improved performance with limited target data.

Conclusion and Future Work

SHDA-RF demonstrates that labels can serve as a universal bridge between heterogeneous feature spaces. By utilizing Random Forests as natural clusterers and enforcing sparsity through LASSO, the method achieves robust semantic alignment without requiring direct correspondence between data points. Future research will explore multi-source domain adaptation and techniques to explicitly address instance-level distribution shifts where marginal or conditional probability distributions differ.

Summary updated

BibTeX
@inproceedings{sukhija2016supervised,
  author = {Sanatan Sukhija and Narayanan Chatapuram Krishnan and Gurkanwal Singh},
  title = {{Supervised Heterogeneous Domain Adaptation via Random Forests}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure