Locality Preserving Projection for Domain Adaptation with Multi-Objective Learning

Le Shu, Tianyang Ma, Longin Jan LateckiAAAI 2014 · DOI

Abstract

Domain adaptation aims to leverage knowledge from a labeled source domain to improve performance on an unlabeled target domain. A primary challenge in unsupervised domain adaptation is the selection of trade-off parameters without target labels. This paper proposes Locality Preserving Projection for Domain Adaptation (LPP-DA), which treats the preservation of source and target structures as a multi-objective optimization problem. By identifying the Pareto frontier, the method avoids manual parameter tuning while maintaining discriminative local structures.

Introduction

Machine learning models often suffer from performance degradation when applied to data from a different distribution, a problem known as domain shift. Standard unsupervised domain adaptation techniques attempt to find a shared feature space where source and target distributions align. However, balancing the preservation of source labels with the adaptation to target structures typically requires a trade-off parameter ($\alpha$) that is difficult to tune without target labels. This paper, presented at AAAI 2014, introduces a novel approach to domain adaptation by combining Locality Preserving Projection (LPP) with Multi-Objective Learning to solve this problem.

Methodology

The LPP-DA method follows a three-step process:

  1. Graph Construction: Adjacency graphs are built for both domains. The source graph uses labels to connect same-class points (supervised), while the target graph uses $p$-nearest neighbors based on cosine similarity (unsupervised).
  2. Multi-Objective Formulation: Instead of a weighted sum of costs, the source and target locality-preserving objectives are treated as separate costs to be minimized simultaneously.
  3. Pareto Optimization: The algorithm searches for the Pareto frontier—solutions where improving one objective necessitates degrading the other. This is solved efficiently using generalized eigendecomposition and SVD for numerical stability under mutually orthogonal constraints.

Experimental Setup

The method was evaluated on diverse datasets, including gene expression data and handwritten digits.

The USPS dataset involves training on specific digit pairs (e.g., 1 and 8) to classify others (e.g., 7 and 3), representing a significant domain shift.

Tasks included USPS digit classification (USPS1 and USPS2) and various cancer subtype classifications (Lung, Brain, and cross-organ tasks like Bladder-to-Uterus). These tasks test the model's ability to handle high-dimensional data (up to 15,000 features) and varying sample sizes.

Results and Discussion

LPP-DA consistently outperformed baseline methods, including Transfer Component Analysis (TCA) and Geodesic Flow Kernel (GFK).

Comparison of classification accuracy across various datasets. LPP-DA (Our method) shows superior performance in nearly all scenarios.

Key findings include:

  • Superior Accuracy: On the Lung1 dataset, LPP-DA achieved 81.58% accuracy, significantly higher than GFK (60.53%).
  • Robustness to Dissimilarity: The method excelled on the USPS datasets where distributions were intentionally dissimilar, reaching 90.36% on USPS1 compared to TCA's 56.10%.
  • High-Dimensional Efficacy: The approach remained effective on gene expression data with over 10,000 features despite small sample sizes, demonstrating the power of preserving local manifold structures.

Conclusion

By leveraging Pareto optimization, LPP-DA provides a robust framework for unsupervised domain adaptation that bypasses the need for heuristic parameter tuning. While the current linear mapping is effective, future work may explore non-linear extensions via kernelization and further optimization for large-scale datasets to reduce computational complexity.

Summary updated

BibTeX
@inproceedings{shu2014locality,
  author = {Le Shu and Tianyang Ma and Longin Jan Latecki},
  title = {{Locality Preserving Projection for Domain Adaptation with Multi-Objective Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.9000},
}
Figure