Private Domain Adaptation from a Public Source

Raef Bassily, Mehryar Mohri, Ananda Theertha SuresharXiv 2022 · DOI · arXiv

Introduction

Domain adaptation (DA) is a cornerstone of modern machine learning, allowing models trained on a labeled "source" domain to generalize to a related but distinct "target" domain. This is particularly useful when labeled data is scarce in the target domain but plentiful elsewhere. However, a significant challenge arises when the target domain contains sensitive information—such as medical records or personal financial data.

Standard domain adaptation techniques require analyzing the target distribution to minimize the "discrepancy" between domains. If the target data is private, this process risks leaking sensitive information. This paper addresses this critical gap by introducing the first differentially private (DP) algorithms for domain adaptation where a learner utilizes a public, labeled source dataset to adapt to a private, unlabeled target dataset.

The Challenge of Non-Smoothness

The mathematical foundation of domain adaptation often relies on minimizing the discrepancy between distributions. Traditionally, this discrepancy is measured using the spectral norm of matrices. From a privacy perspective, the spectral norm is problematic: it is non-smooth and possesses high sensitivity. In Differential Privacy, high sensitivity necessitates the addition of significant noise to protect individual data points, which often destroys the model's utility.

To overcome this, the authors introduce a Softmax Smooth Approximation. By replacing the non-smooth spectral norm with a surrogate based on the Log-Sum-Exp of eigenvalues, they create a function that is both differentiable and has low gradient sensitivity. These properties are the essential ingredients required to apply DP noise effectively without sacrificing the ability to optimize the model.

Proposed Private Algorithms

The paper develops two primary optimization frameworks to handle the privatized discrepancy:

  1. Two-Stage Approach: The algorithm first uses a DP mechanism to calculate weights for the source data that minimize the discrepancy to the target. Once these weights are privatized, the subsequent model training on the weighted source samples is "privacy-free," as it no longer directly interacts with the raw private target data.
  2. Single-Stage (Joint) Approach: The authors propose a novel algorithm that simultaneously updates the model weights and the predictor. This joint optimization often proves more effective in practice.

Within these frameworks, the authors provide "Noisy" variants of two classic optimization algorithms:

  • Noisy Frank-Wolfe: Theoretically superior when the number of target samples ($n$) is large, as its error scales at a rate of $\tilde{O}(1/(\epsilon n)^{1/3})$.
  • Noisy Mirror Descent: More efficient in terms of the number of source samples ($m$), though it can be more sensitive to the injected privacy noise.

Experimental Evaluation

The algorithms were evaluated using synthetic 10-dimensional datasets designed to simulate distribution shifts. The target domain was modeled as a spherical Gaussian, while the source domain consisted of a mixture of Gaussians (one matching the target and one shifted). The task was a regression problem using a piecewise linear "hinge" labeling function.

Key Findings

  • Privacy-Utility Trade-off: The experiments demonstrate that utility (measured by test error) degrades gracefully as the privacy budget ($\epsilon$) becomes stricter. Even under high privacy constraints, the adaptation significantly outperforms models trained solely on source data.
  • Algorithm Performance: In head-to-head empirical tests, the single-stage Noisy Frank-Wolfe algorithm consistently achieved the lowest test error across various privacy levels and sample sizes.
  • Scalability: The methods proved robust as the number of unlabeled target samples increased from 1,000 to 8,000, confirming the theoretical convergence rates.

Conclusion and Future Work

This research provides a rigorous framework for performing domain adaptation under the constraints of Differential Privacy. By identifying that smoothness is the gateway to privacy in distribution matching, the authors have opened a new path for sensitive data utilization.

While the current work focuses on linear hypothesis classes and synthetic data, it establishes the necessary theoretical bounds for future expansions. Subsequent research may look toward extending these DP guarantees to deep neural networks and complex, high-dimensional real-world datasets where distribution shifts are more irregular.

Summary updated

Abstract

A key problem in a variety of applications is that of domain adaptation from a public source domain, for which a relatively large amount of labeled data with no privacy constraints is at one's disposal, to a private target domain, for which a private sample is available with very few or no labeled data. In regression problems with no privacy constraints on the source or target data, a discrepancy minimization algorithm based on several theoretical guarantees was shown to outperform a number of other adaptation algorithm baselines. Building on that approach, we design differentially private discrepancy-based algorithms for adaptation from a source domain with public labeled data to a target domain with unlabeled private data. The design and analysis of our private algorithms critically hinge upon several key properties we prove for a smooth approximation of the weighted discrepancy, such as its smoothness with respect to the $\ell_1$-norm and the sensitivity of its gradient. Our solutions are based on private variants of Frank-Wolfe and Mirror-Descent algorithms. We show that our adaptation algorithms benefit from strong generalization and privacy guarantees and report the results of experiments demonstrating their effectiveness.
BibTeX
@misc{bassily2022private,
  author = {Raef Bassily and Mehryar Mohri and Ananda Theertha Suresh},
  title = {{Private Domain Adaptation from a Public Source}},
  year = {2022},
  eprint = {2208.06135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.06135},
}
Figure