Heterogeneous Domain Adaptation Using Manifold Alignment

Chang Wang, Sridhar MahadevanIJCAI 2011 · OpenReview

Abstract

We propose a manifold alignment based approach for heterogeneous domain adaptation. A key aspect of this approach is to construct mappings to link different feature spaces in order to transfer knowledge across domains. The new approach can reuse labeled data from multiple source domains in a target domain even in the case when the input domains do not share any common features or instances. As a pre-processing step, our approach can also be combined with existing domain adaptation approaches to learn a common feature space for all input domains. This paper extends existing manifold alignment approaches by making use of labels rather than correspondences to align the manifolds. This extension significantly broadens the application scope of manifold alignment, since the correspondence relationship required by existing alignment approaches is hard to obtain in many applications.

Introduction

Standard Domain Adaptation assumes that source and target data reside in the same feature space but follow different distributions. However, Heterogeneous Domain Adaptation (HDA) addresses a more complex scenario where feature spaces are entirely different—for instance, transferring knowledge from English text to Italian text, or from text descriptions to images.

Existing manifold alignment techniques often require instance-level correspondence, meaning one must know exactly which data point in the source domain matches a specific point in the target domain. This paper introduces a method that bypasses this requirement by using class labels as a "universal translator" to align disparate manifolds into a shared latent space.

Methodology

The proposed framework projects multiple disparate data sources into a single, shared latent space. Unlike previous methods that rely on paired instances, this approach uses semantic labels to bridge the gap between domains.

Conceptual framework of the proposed manifold alignment approach for heterogeneous domain adaptation.

The algorithm optimizes a cost function with three primary objectives:

  1. Class Similarity: Projecting instances with the same label close together, regardless of their original domain.
  2. Class Separation: Pushing instances with different labels far apart.
  3. Topology Preservation: Maintaining the internal geometric structure (manifold) of each individual domain.

Mathematically, this is solved as a generalized eigenvalue problem: $Z(\mu L + L_s)Z^T x = \lambda ZL_d Z^T x$. This solution yields optimal linear mapping functions that transform data from high-dimensional original spaces into the aligned latent space, where standard classifiers can then be deployed.

Experimental Results

The approach was evaluated across three distinct domains: Protein Tertiary Structure (for visualization), the TDT2 Corpus (text categorization), and the TREC Collection (learning to rank).

In the TDT2 experiments, the authors simulated a heterogeneous environment by representing document subsets using different feature sets (LSI vs. LDA). The results demonstrated that label-based manifold alignment consistently outperformed traditional methods like Canonical Correlation Analysis (CCA) and standard manifold regularization.

Performance comparison showing the proposed label-based manifold alignment outperforming baseline methods.

Key findings include:

  • Superior Accuracy: The method achieved higher precision by effectively leveraging the shared semantic structure of labels.
  • Robustness: The inclusion of a manifold regularization term ($\mu L$) ensures that the "shape" of the data is preserved, allowing the model to utilize unlabeled target data to improve alignment.
  • Visualization: Qualitative tests on protein structures showed the algorithm could successfully rescale and align distorted datasets, bringing similar classes into proximity while maintaining the data's natural flow.

Conclusion and Future Work

By replacing instance correspondence with label-based alignment, this method significantly expands the utility of manifold alignment in real-world HDA tasks. While the current work focuses on linear mappings and requires some labeled data in the target domain, future research could explore non-linear kernel extensions and more computationally efficient approximations for large-scale datasets.

Summary updated

Abstract

We propose a manifold alignment based approach for heterogeneous domain adaptation. A key aspect of this approach is to construct mappings to link different feature spaces in order to transfer knowledge across domains. The new approach can reuse labeled data from multiple source domains in a target domain even in the case when the input domains do not share any common features or instances. As a pre-processing step, our approach can also be combined with existing domain adaptation approaches to learn a common feature space for all input domains. This paper extends existing manifold alignment approaches by making use of labels rather than correspondences to align the manifolds. This extension significantly broadens the application scope of manifold alignment, since the correspondence relationship required by existing alignment approaches is hard to obtain in many applications.
BibTeX
@inproceedings{wang2011heterogeneous,
  author = {Chang Wang and Sridhar Mahadevan},
  title = {{Heterogeneous Domain Adaptation Using Manifold Alignment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=HyWNwHM_Zr},
}
Figure