Introduction
Domain adaptation (DA) is a cornerstone of modern machine learning, allowing models trained on a labeled "source" domain to generalize to a related but distinct "target" domain. This is particularly useful when labeled data is scarce in the target domain but plentiful elsewhere. However, a significant challenge arises when the target domain contains sensitive information—such as medical records or personal financial data.
Standard domain adaptation techniques require analyzing the target distribution to minimize the "discrepancy" between domains. If the target data is private, this process risks leaking sensitive information. This paper addresses this critical gap by introducing the first differentially private (DP) algorithms for domain adaptation where a learner utilizes a public, labeled source dataset to adapt to a private, unlabeled target dataset.
The Challenge of Non-Smoothness
The mathematical foundation of domain adaptation often relies on minimizing the discrepancy between distributions. Traditionally, this discrepancy is measured using the spectral norm of matrices. From a privacy perspective, the spectral norm is problematic: it is non-smooth and possesses high sensitivity. In Differential Privacy, high sensitivity necessitates the addition of significant noise to protect individual data points, which often destroys the model's utility.
To overcome this, the authors introduce a Softmax Smooth Approximation. By replacing the non-smooth spectral norm with a surrogate based on the Log-Sum-Exp of eigenvalues, they create a function that is both differentiable and has low gradient sensitivity. These properties are the essential ingredients required to apply DP noise effectively without sacrificing the ability to optimize the model.
Proposed Private Algorithms
The paper develops two primary optimization frameworks to handle the privatized discrepancy:
- Two-Stage Approach: The algorithm first uses a DP mechanism to calculate weights for the source data that minimize the discrepancy to the target. Once these weights are privatized, the subsequent model training on the weighted source samples is "privacy-free," as it no longer directly interacts with the raw private target data.
- Single-Stage (Joint) Approach: The authors propose a novel algorithm that simultaneously updates the model weights and the predictor. This joint optimization often proves more effective in practice.
Within these frameworks, the authors provide "Noisy" variants of two classic optimization algorithms:
- Noisy Frank-Wolfe: Theoretically superior when the number of target samples ($n$) is large, as its error scales at a rate of $\tilde{O}(1/(\epsilon n)^{1/3})$.
- Noisy Mirror Descent: More efficient in terms of the number of source samples ($m$), though it can be more sensitive to the injected privacy noise.
Experimental Evaluation
The algorithms were evaluated using synthetic 10-dimensional datasets designed to simulate distribution shifts. The target domain was modeled as a spherical Gaussian, while the source domain consisted of a mixture of Gaussians (one matching the target and one shifted). The task was a regression problem using a piecewise linear "hinge" labeling function.
Key Findings
- Privacy-Utility Trade-off: The experiments demonstrate that utility (measured by test error) degrades gracefully as the privacy budget ($\epsilon$) becomes stricter. Even under high privacy constraints, the adaptation significantly outperforms models trained solely on source data.
- Algorithm Performance: In head-to-head empirical tests, the single-stage Noisy Frank-Wolfe algorithm consistently achieved the lowest test error across various privacy levels and sample sizes.
- Scalability: The methods proved robust as the number of unlabeled target samples increased from 1,000 to 8,000, confirming the theoretical convergence rates.
Conclusion and Future Work
This research provides a rigorous framework for performing domain adaptation under the constraints of Differential Privacy. By identifying that smoothness is the gateway to privacy in distribution matching, the authors have opened a new path for sensitive data utilization.
While the current work focuses on linear hypothesis classes and synthetic data, it establishes the necessary theoretical bounds for future expansions. Subsequent research may look toward extending these DP guarantees to deep neural networks and complex, high-dimensional real-world datasets where distribution shifts are more irregular.
Summary updated