Many deployed classifiers are trained on a public, labeled source dataset, such as product photos or a public hospital archive, and then used on a different target population whose labels are sensitive, such as a new clinic's patients or a set of user devices. Adapting across this gap without any target labels is the job of adversarial domain adaptation. These methods train a feature extractor to fool a second network, a "domain discriminator," that tries to tell source features from target features. Even the best such methods leave large errors when the shift is hard. Asking target users for labels would fix much of this, but only if their labels stay private. The standard tool for that is randomized response: each user reports their true label with some probability and a random wrong one otherwise. This gives plausible deniability with a strength set by a privacy budget $\epsilon$ (smaller means more private and noisier). Randomized response becomes far more useful when it is given a good guess, or "prior," about which labels are plausible for each example. The prior lets the mechanism randomize only among a few likely candidates instead of all classes.
The proposal joins these two ideas. A model adapted with no target labels at all is exactly a good, privacy-free prior over target labels. It is built only from public source data and unlabeled target inputs, so using it costs no privacy budget. We feed that adapted model's predictions into prior-guided randomized response on the target users. We then feed the resulting private labels back into the adversarial alignment, which corrects the class confusions that unsupervised alignment gets wrong. The claim is that this loop gives much higher target accuracy than either piece alone, especially at strong privacy ($\epsilon \le 2$), where label-private learning from scratch is weakest. We also expect the gain to be predictable from how often the adapted model's top few guesses contain the true label.
Core Claim and How It Differs From Prior Work
Two existing lines of work supply the method. Deep Learning with Label Differential Privacy introduces RRWithPrior, which restricts randomized response to the top-$k$ classes under a prior, and LP-MST, a multi-stage scheme in which priors come from models trained on earlier private stages or from self-supervised clustering. It reaches 94.7% on CIFAR-10 at $\epsilon=8$. Its first stage, however, has no informative prior, and it assumes training and test data come from the same distribution. Conditional Adversarial Domain Adaptation (CDAN) conditions the domain discriminator on the classifier's predicted class probabilities, combined with features through a multilinear map. It adds entropy weighting (CDAN+E) to emphasize confident examples. It assumes clean source labels and no target labels, and it inherits the target-side errors of its own predictions.
Our proposal differs from both. Unlike LP-MST, the prior for the very first private stage comes from cross-domain adaptation and costs zero privacy budget. Unlike CDAN, the target-side conditioning uses a posterior that combines the model's prediction with each user's private report.
One idea at this intersection might seem obvious: privatize source labels and plug a known noise matrix into CDAN. Two facts make it less interesting than it looks. First, CDAN conditions on predictions rather than labels, so source label noise reaches the discriminator only through the classifier, and standard forward loss correction already addresses that. Second, Label-Noise Robust Domain Adaptation (DCIC) already debiases distribution matching with a known transition matrix, albeit with kernel moment matching rather than a discriminator. It reports 68.50% versus 54.76% for DAN on MNIST→USPS at 40% noise. DCIC therefore constrains our design. It tells us that a known noise matrix should be used explicitly rather than treated as unknown corruption, and it leaves little room for a source-side contribution. We consequently put the private labels on the target side, where no prior work in the file operates.
The quantitative handle comes from RRWithPrior itself. Suppose the prior's top-$k$ set contains the true label with probability $c_k$, which we call "prior coverage." The mechanism then outputs the true label with probability
$$ P(\tilde y = y) = c_k \cdot \frac{e^{\epsilon}}{e^{\epsilon} + k - 1}, $$
where $\tilde y$ is the reported label. The first factor is the chance the truth survives the restriction. The second is $k$-ary randomized response on the restricted set (Kairouz et al.). Without a prior, $k$ equals the number of classes $K$, so on 65-class Office-Home at $\epsilon=1$ the true label is reported with probability under 5%. If an adapted model attains coverage $c_3 \approx 0.85$, the same budget yields roughly 0.47. Adaptation raises $c_k$ at small $k$. That increase is the entire mechanism of gain, and it is measurable before any private training.
Research Program
The system runs in three steps.
Step 1: unsupervised adaptation. We adapt from source to target with CDAN+E, using no target labels. This yields predicted class probabilities $g(x)$ for every target input.
Step 2: private label collection. Each target user receives the prior $g(x)$ for their own input, which is computed without any private label. The user privatizes their label with RRWithPrior, choosing $k$ per example to maximize the expression above under the prior's own estimate of $c_k$, as Ghazi et al. do.
Step 3: joint training. We continue joint training with three losses.
- Source loss. The ordinary classification loss on source data.
- Target loss. A loss on the private target reports that is forward-corrected through the exact, known matrix $P(\tilde y \mid y)$ of the mechanism used for each example.
- Alignment loss. An alignment term in which the discriminator's target-side conditioning vector is replaced by the label posterior $\pi(y \mid x, \tilde y) \propto g(x)_y \, P(\tilde y \mid y)$.
The posterior is sharper than $g$ wherever the report disagrees with a confident-but-wrong prediction. This attacks the "cars mapped to trucks" mode mismatch that CDAN was built to reduce but cannot fully remove without labels. The same posterior's entropy replaces CDAN+E's prediction entropy in the example weighting. This avoids a failure mode we expect, though no paper in the file has demonstrated it: entropy weighting on predictions trained against noisy targets would downweight exactly the examples whose private reports carry the most correction.
Multiple rounds and the privacy argument. Following LP-MST, target users can be split into disjoint groups queried in successive rounds, each round using the improved adapted model as prior. Privacy follows from three facts. Each user is queried once. Their prior depends only on public data, unlabeled features, and other users' already-privatized labels. RRWithPrior is $\epsilon$-label-DP for any such prior. This is the label-only privacy model of Ghazi et al., so input features are treated as non-sensitive, a limitation those authors also note.
Many-class variant. For large label spaces we also test the bit-vector randomization of Zhao et al. as the target mechanism. That paper reports 64.4% on CIFAR-100 at $\epsilon=2$. Because its sigmoid outputs lose the sum-to-one structure that CDAN's conditioning and entropy rely on, we renormalize the sigmoid outputs before conditioning. We report whether this works rather than assume it.
Analysis. The analysis component checks the coverage relation empirically. For each benchmark and $\epsilon$, we plot final target accuracy against the measured coverage $c_k$ of the step-1 model. The prediction is that gain over a no-prior baseline tracks $c_k \cdot e^{\epsilon}/(e^{\epsilon}+k-1)$.
Baselines
Unsupervised CDAN+E. This is the naive reference, or "do nothing with private labels," point. It is the strongest existing method that uses no target labels, and our method must beat it to justify asking users for anything. The paper reports 87.7% average accuracy on Office-31 and 65.8% on Office-Home with a ResNet-50 backbone (Conditional Adversarial Domain Adaptation). We rerun it in our codebase with the same backbone and schedule, and we first confirm we land near those numbers. Beating it means a statistically clear gain at $\epsilon \le 2$ across seeds.
LP-MST on the target users alone. This is the strongest existing label-private learner. We initialize from the source-only model, use the same users and budget, use LP-MST's own stage-1 prior (uniform or self-supervised clustering), and apply no adversarial alignment. Its reported numbers are on CIFAR (74.14% on CIFAR-100). We found no reported result on Lacuna for LP-MST or any label-DP method on Office-31, Office-Home, or other shifted benchmarks, so we run it ourselves.
RRWithPrior using a source-only prior. This baseline is identical to ours except that the prior comes from a model trained on source alone, with no adversarial step anywhere. It isolates whether adaptation, rather than merely having some cross-domain prior, is what raises coverage.
Our full method with plain $g$ conditioning. This ablation keeps everything in our method but conditions the discriminator on the plain prediction $g$ instead of the posterior. It isolates the second key ingredient.
The two extremes. For context we include a central DP-SGD source model followed by CDAN, and a non-private upper bound that trains with true target labels.
Fairness rules. All methods get the same target users, the same total budget $\epsilon$ per user, the same backbone, epochs, and tuning grid. Hyperparameters are selected on a held-out source split or on the private reports themselves, never on clean target labels, which a real deployment would not have.
Why not just…?
Why not just run unsupervised adaptation and skip private labels? No privacy is spent, and CDAN+E is already strong on Office-31. But its 65.8% on Office-Home leaves a large gap to supervised training. Unsupervised alignment cannot detect a consistent class swap, because a swap still produces aligned distributions. This alternative is a real threat at very small $\epsilon$, where even a good prior yields few informative reports. Our $\epsilon$ sweep will locate the budget below which private labels stop paying for themselves, and we will report that crossover honestly.
Why not just use LP-MST with its clustering prior on the target? Ghazi et al. already supply a privacy-free prior from self-supervised clustering. However, clusters carry no class names. A cluster-derived prior narrows the candidates only after private labels have been collected to name the clusters. The adapted model's prior is class-named from the start because it inherits names from the source labels. If clustering priors match adaptation priors in measured coverage $c_k$, our main claim fails. The coverage plot settles this directly.
Why not just fine-tune the source model on target RR labels with noise correction, DCIC-style, with no adversary? Known-matrix forward correction is cheap and principled, and DCIC shows it works under shift (Label-Noise Robust Domain Adaptation). What it lacks is a good prior and any use of the many unlabeled target inputs beyond those that were queried. The source-only-prior baseline and the plain-$g$ ablation are exactly this alternative and its near neighbor. If they match the full method, the adversarial component is unnecessary and we will say so.
Why not just train the source model with central DP-SGD? That approach protects a different party, a trusted server's source data, and Ghazi et al. report that DP-SGD struggles to exceed 70% on CIFAR-10 where label-DP reaches 94.7%. It answers a different threat model, so we include it only for context.
Proposed Experiments
Benchmarks and budgets. We use Office-31, Office-Home, and MNIST→USPS, with VisDA-2017 as a scale test. Office-Home's 65 classes are where restriction to a small top-$k$ matters most. Privacy budgets are $\epsilon \in \{0.5, 1, 2, 4, 8\}$. We report target accuracy over five seeds, the step-1 coverage $c_k$, and the realized fraction of correct reports.
Privacy audit. As a sanity check rather than a guarantee, we run the Bayesian label-inference attack of Auditing Privacy Mechanisms via Label Inference Attacks against the final model. We measure additive and multiplicative reconstruction advantage, and we check whether adversarial alignment changes empirical leakage relative to LP-MST at the same $\epsilon$. That paper audited only single-domain settings.
Decisive checks. The proposal succeeds if three things hold. First, at $\epsilon \le 2$ on Office-Home it beats both unsupervised CDAN+E and LP-MST. Second, it beats the source-only-prior variant by a margin that the measured coverage difference predicts. Third, the posterior-conditioning ablation shows a gain concentrated on the class pairs that unsupervised CDAN confuses most.
Pre-registered risk. We name one risk in advance. Adaptation errors are correlated: a mode-mismatched class may have its true label outside the top-$k$ for most of its examples. RRWithPrior then cannot recover that class, and later rounds may entrench the error. We will report per-class coverage to detect this. We will test whether the adaptive choice of $k$ and multi-round re-querying recover such classes, and we will treat persistent failure as a limit of the approach rather than tune around it.
References
- Deep Learning with Label Differential Privacy
- Conditional Adversarial Domain Adaptation
- Label-Noise Robust Domain Adaptation
- Discrete Distribution Estimation under Local Privacy
- Enhancing Learning with Label Differential Privacy by Vector Approximation
- Auditing Privacy Mechanisms via Label Inference Attacks