Abstract
Domain adaptation aims to leverage knowledge from a labeled source domain to improve performance on an unlabeled target domain. A primary challenge in unsupervised domain adaptation is the selection of trade-off parameters without target labels. This paper proposes Locality Preserving Projection for Domain Adaptation (LPP-DA), which treats the preservation of source and target structures as a multi-objective optimization problem. By identifying the Pareto frontier, the method avoids manual parameter tuning while maintaining discriminative local structures.
Introduction
Machine learning models often suffer from performance degradation when applied to data from a different distribution, a problem known as domain shift. Standard unsupervised domain adaptation techniques attempt to find a shared feature space where source and target distributions align. However, balancing the preservation of source labels with the adaptation to target structures typically requires a trade-off parameter ($\alpha$) that is difficult to tune without target labels. This paper, presented at AAAI 2014, introduces a novel approach to domain adaptation by combining Locality Preserving Projection (LPP) with Multi-Objective Learning to solve this problem.
Methodology
The LPP-DA method follows a three-step process:
- Graph Construction: Adjacency graphs are built for both domains. The source graph uses labels to connect same-class points (supervised), while the target graph uses $p$-nearest neighbors based on cosine similarity (unsupervised).
- Multi-Objective Formulation: Instead of a weighted sum of costs, the source and target locality-preserving objectives are treated as separate costs to be minimized simultaneously.
- Pareto Optimization: The algorithm searches for the Pareto frontier—solutions where improving one objective necessitates degrading the other. This is solved efficiently using generalized eigendecomposition and SVD for numerical stability under mutually orthogonal constraints.
Experimental Setup
The method was evaluated on diverse datasets, including gene expression data and handwritten digits.
Tasks included USPS digit classification (USPS1 and USPS2) and various cancer subtype classifications (Lung, Brain, and cross-organ tasks like Bladder-to-Uterus). These tasks test the model's ability to handle high-dimensional data (up to 15,000 features) and varying sample sizes.
Results and Discussion
LPP-DA consistently outperformed baseline methods, including Transfer Component Analysis (TCA) and Geodesic Flow Kernel (GFK).
Key findings include:
- Superior Accuracy: On the Lung1 dataset, LPP-DA achieved 81.58% accuracy, significantly higher than GFK (60.53%).
- Robustness to Dissimilarity: The method excelled on the USPS datasets where distributions were intentionally dissimilar, reaching 90.36% on USPS1 compared to TCA's 56.10%.
- High-Dimensional Efficacy: The approach remained effective on gene expression data with over 10,000 features despite small sample sizes, demonstrating the power of preserving local manifold structures.
Conclusion
By leveraging Pareto optimization, LPP-DA provides a robust framework for unsupervised domain adaptation that bypasses the need for heuristic parameter tuning. While the current linear mapping is effective, future work may explore non-linear extensions via kernelization and further optimization for large-scale datasets to reduce computational complexity.
Summary updated