TL;DR
This paper introduces PRIMA, the first method to achieve differentially private correlation alignment for domain adaptation. PRIMA protects the privacy of both source and target data by introducing noise during covariance estimation and model training, while novel techniques like random subspace ensembles and shrinking-based matrix recovery mitigate privacy-induced performance degradation, especially in high-dimensional settings.
What Problem Does This Paper Address?
Domain adaptation (DA) is a crucial technique in machine learning that allows models trained on abundant data from a "source" domain to perform well on a related "target" domain with limited or no labeled data. A popular DA method is Correlation Alignment (CORAL), which aligns the statistical distributions of the source and target domains by matching their covariance matrices. However, in many real-world scenarios, such as collaborations between different institutions, sharing raw data or even statistical summaries like covariance matrices poses significant privacy risks. Existing privacy-preserving DA methods often have limitations, like requiring shared features or being sensitive to data distribution mismatches. This paper addresses the critical need for a domain adaptation method that provides strong, verifiable privacy guarantees (differential privacy) without sacrificing performance, especially when dealing with high-dimensional data and when source and target domains are unwilling to trust each other.
Methods
PRIMA tackles the challenge of privacy-preserving domain adaptation through a three-phase approach:
- RSE-based Covariance Estimation: To protect the target domain's covariance matrix, PRIMA uses a Random Subspace Ensemble (RSE) approach. Instead of perturbing the entire high-dimensional covariance matrix directly (which leads to large errors), the target domain's feature space is split into several lower-dimensional subspaces. Gaussian noise is then added to the covariance matrices of these smaller subspaces. This strategy leverages the composition property of differential privacy to manage the privacy budget more effectively.
- Positive Semi-definite Matrix Recovery: Perturbing covariance matrices can destroy their positive semi-definite property, which is essential for many machine learning algorithms. PRIMA introduces a "shrinking-based" method to recover this property. It finds the nearest positive semi-definite matrix to the perturbed one by forming a convex combination and using a bisection method to find an optimal shrinking parameter. This ensures the recovered matrix is valid and, importantly, provides better utility than the unrecovered perturbed matrix.
- Model Training with Differential Privacy: For the source domain, PRIMA first generates subspaces corresponding to the target domain's splitting. It then performs correlation alignment on each subspace. To ensure privacy during model training, PRIMA employs the differentially private mechanism proposed in [Abadi et al., 2016], which involves perturbing gradients with Gaussian noise during stochastic gradient descent. The moments accountant theorem is used to provide tight privacy bounds.
A key innovation is the use of a forward selection strategy to determine the optimal dimension for each subspace in the RSE phase, balancing information loss and noise impact.
Datasets
The paper evaluates PRIMA on two standard domain adaptation benchmark datasets:
- Office-Caltech10: This dataset contains 10 object categories across four domains (Webcam, DSLR, Amazon, Caltech256), resulting in 12 domain adaptation tasks. Features are encoded into 4096-dimensional vectors.
- Amazon review dataset: This dataset consists of Amazon reviews from four domains (Book, DVD, Kitchen, Electronics), with 1000 positive and 1000 negative reviews per domain. Reviews are encoded into 5,000-dimensional feature vectors using a bag-of-word method, and four source-target pairs are used for adaptation.
Key Insights and Ideas
- Subspace Decomposition is Key for High-Dimensional Privacy: Directly perturbing high-dimensional covariance matrices leads to poor utility due to the large error bounds. Splitting into low-dimensional subspaces (RSE) is a crucial step to manage noise and improve privacy-utility trade-offs.
- Preserving Positive Semi-Definiteness Matters: The positive semi-definite property of covariance matrices is not just a theoretical nicety; its destruction significantly degrades performance. The shrinking-based recovery method is essential for making the privacy-preserving covariance estimation practical.
- Source and Target Privacy are Both Achievable: PRIMA demonstrates that it's possible to protect both the source and target domains' data privacy simultaneously, enabling trustless collaboration.
- Forward Selection for Optimal Subspace Dimensions: Automatically determining subspace dimensions using forward selection is a novel way to balance the trade-off between capturing subspace information and managing noise, leading to better performance.
Key Results
- PRIMA achieves (ε, δ)-differential privacy for both source and target data.
- Experimental results on Office-Caltech10 and Amazon review datasets show that PRIMA and its variant PRISA (using SA instead of CORAL) outperform existing differentially private domain adaptation methods (DPDA, GDPDA) by a significant margin (e.g., 1-2% accuracy improvement).
- PRIMA and PRISA show only a ~1% drop in accuracy compared to non-private methods, while DPDA and GDPDA drop by 3-5%, highlighting PRIMA's superior utility.
- Ablation studies confirm the effectiveness of each component: RSE-based covariance estimation, forward selection for subspace generation, and the shrinking-based PSD matrix recovery all contribute to improved performance.
Limitations and Future Work
- Privacy Budget Allocation: While PRIMA uses parallel composition for privacy, the paper doesn't explicitly detail how the privacy budget is allocated across different components (covariance estimation, model training) for optimal trade-offs.
- Computational Cost: The RSE approach and forward selection might increase computational complexity compared to simpler methods, although it's presented as more efficient than some deep adaptation approaches.
- Future Work: The paper suggests extending PRIMA to other domain adaptation methods and exploring more advanced privacy mechanisms. Further investigation into the optimal partitioning strategies for RSE and more efficient PSD matrix recovery techniques could also be beneficial.
Summary updated