Abstract
Transfer learning addresses the problem of how to utilize plenty of labeled data in a source domain to solve related but different problems in a target domain, even when the training and testing problems have different distributions or features. In this paper, we consider transfer learning via dimensionality reduction. To solve this problem, we learn a low-dimensional latent feature space where the distributions between the source domain data and the target domain data are the same or close to each other. Onto this latent feature space, we project the data in related domains where we can apply standard learning algorithms to train classification or regression models. Thus, the latent feature space can be treated as a bridge of transferring knowledge from the source domain to the target domain. The main contribution of our work is that we propose a new dimensionality reduction method to find a latent space, which minimizes the distance between distributions of the data in different domains in a latent space. The effectiveness of our approach to transfer learning is verified by experiments in two real world applications: indoor WiFi localization and binary text classification.
Introduction
In traditional machine learning, a fundamental assumption is that training and testing data are drawn from the same distribution. However, in many real-world scenarios, this assumption is violated. For instance, WiFi signal strength data used for indoor localization can vary significantly depending on the time of day or environmental changes. When the distribution of the target domain shifts away from the source domain, models trained on source data typically suffer a significant drop in performance.
Transfer learning aims to bridge this gap. While previous approaches have focused on instance re-weighting or feature selection, this paper proposes a dimensionality reduction framework that explicitly aligns the distributions of different domains to create a shared representation.
Maximum Mean Discrepancy Embedding (MMDE)
The core of our approach is Maximum Mean Discrepancy Embedding (MMDE). The goal is to find a low-dimensional latent space that acts as a "bridge" between domains. If the source and target data appear to come from the same distribution in this latent space, knowledge can be transferred seamlessly.
Methodology
The MMDE process involves finding a kernel matrix through a Semidefinite Program (SDP). The optimization objective is three-fold:
- Minimize Distribution Distance: We use Maximum Mean Discrepancy (MMD), a non-parametric metric, to measure the distance between the source and target distributions in a Reproducing Kernel Hilbert Space (RKHS).
- Maximize Variance: To avoid trivial solutions where all data points collapse to a single point, we maximize the total variance of the data, a technique borrowed from Maximum Variance Unfolding (MVU).
- Preserve Local Geometry: We ensure that the local neighborhood structure of the original data is maintained in the latent space.
As shown in the synthetic data comparison above, standard dimensionality reduction techniques like MVU preserve the separation between different distributions. In contrast, MMDE successfully aligns the source and target data in the 1D latent space, making them indistinguishable for downstream learners.
Experimental Evaluation
We evaluated MMDE across three distinct domains: synthetic data, indoor WiFi localization, and text classification.
Indoor WiFi Localization
In this regression task, we used Received Signal Strength (RSS) values. The source data was collected at midnight, while the target data was collected two days later in the afternoon. Due to environmental changes, the signal distributions differed significantly.
The results demonstrate that regression models (SVR, RLSR, LapRLSR) trained in the MMDE latent space significantly outperform those trained in the original feature space. The cumulative probability of achieving a low error distance is much higher when using the aligned latent features, proving the effectiveness of the "bridge" concept.
Text Classification
Using the Reuters-21578 dataset, we performed binary classification where source and target domains consisted of different sub-categories. MMDE consistently improved accuracy across the board. For example, in a "people vs. places" task, accuracy increased from 51.9% in the original space to 65.4% in the MMDE latent space using a standard SVM.
Conclusion
MMDE provides a powerful framework for transfer learning by finding a distribution-aligned latent space. By combining MMD minimization with variance maximization, we create a representation where source-trained models remain effective on target data. While the current SDP-based optimization is computationally intensive, future work will focus on more efficient solvers and extending the framework to handle shifts in conditional distributions.
Summary updated