Abstract
Domain Adaptation (DA) is a powerful technique for training models on a target domain with limited data by leveraging a related source domain. However, when target data is sensitive, standard DA methods fail to provide privacy protections. This paper introduces the first differentially private (DP) algorithms for supervised domain adaptation with rigorous theoretical convergence guarantees. By employing a novel reparameterization of sample weights, we demonstrate how to use public source data to enhance the accuracy of models trained on private target data without compromising individual privacy.
Introduction
The tension between data utility and privacy is a central challenge in modern machine learning. While Domain Adaptation (DA) allows models to "borrow" knowledge from a source domain to improve performance on a target domain, it often ignores the privacy risks associated with sensitive target datasets. Conversely, standard Differential Privacy (DP) mechanisms are not optimized for cross-domain knowledge transfer.
This work addresses this gap by proposing two $(\epsilon, \delta)$-DP algorithms: CnvxAdap for linear regression and NCnvxAdap for non-convex models like neural networks. These algorithms are designed to minimize a discrepancy-based objective while adhering to strict privacy constraints.
Methodology
The core of our approach lies in managing the optimization landscape and gradient sensitivity.
Reparameterization and Convexity
For linear regression, the standard discrepancy minimization objective is often non-convex. We introduce a reparameterization trick, transforming sample weights ($u_i = 1/q_i$) to achieve a joint convex optimization. This transformation not only simplifies the optimization but also reduces the sensitivity of the gradients, which is critical for minimizing the noise added for differential privacy.
Smoothing for Non-Convex Models
For complex models, we employ a $μ$-softmax approximation to smooth the objective function. This allows us to provide convergence guarantees to stationary points, ensuring that the model can be trained effectively even when the underlying loss surface is non-convex.
Theoretical Guarantees
We prove that our algorithms achieve a "privacy tax" (excess error) that scales at a nearly optimal rate: $$O\left(\frac{\sqrt{d \log(1/\delta)}}{\epsilon n}\right)$$ where $n$ is the number of private target samples. This result matches known lower bounds for private learning, indicating that our method is theoretically optimal in its use of private data.
Experimental Results
We evaluated our algorithms on a variety of tasks, including regression (UCI datasets), sentiment analysis (multi-domain reviews), and classification (CIFAR, SVHN, ImageNet).
Key Insights
- Superiority over Baselines: Our methods consistently outperformed non-private baselines like Discrepancy Minimization (DM) and Kernel Mean Matching (KMM).
- Privacy at Low Cost: The performance gap between private and non-private versions of CnvxAdap narrows significantly as the privacy budget ($\epsilon$) or sample size ($n$) increases.
- The "Buffer" Effect: Using public source data allows for high accuracy even with a small privacy budget. The model learns general features from the public source, requiring fewer sensitive target samples to fine-tune for the specific domain.
Conclusion
This work provides the first theoretically grounded framework for private domain adaptation. By leveraging public data through a reparameterized optimization lens, we can build robust models that respect individual privacy. Future research will focus on automated hyperparameter tuning and adapting the framework for domains with disparate feature spaces.
Summary updated