Adversarial domain adaptation trains a feature extractor to fool a second network, the "domain discriminator," that tries to tell labeled source examples from unlabeled target examples. When the discriminator can no longer tell them apart, a classifier trained on source labels is expected to work on the target domain. In practice the method often fails in two ways. It merges classes together in feature space, and it overfits the source labels, which the target domain then exposes. Information-theoretic generalization bounds address the second failure. They limit test error by how much the trained model "remembers" about its specific training examples, and a recent class-wise version shows this memorization can differ by a factor of four between classes. Our proposal is to build a per-class target-error bound for adversarial adaptation that adds two measurable terms: how much the model memorized about each source class, and how far apart source and target remain within that class. We then use this bound to set how hard the adversary pushes on each class, in place of the single global strength used today.
The opportunity is that both failure modes act class by class, yet every current tool measures them in aggregate. Domain-Adversarial Neural Networks apply one reversal strength to all classes. The information-theoretic analysis of domain adaptation bounds memorization holistically. The class-wise bounds were derived only for a single domain. If the per-class bound ranks which classes will fail on the target, it becomes a usable training signal and not just a diagnostic. That would also answer a stated open problem in the information-theoretic literature: how to turn these bounds into design rules.
Core Claim
For class $y$, let $\mathcal{R}_T(y)$ be target error and $\widehat{\mathcal{R}}_S(y)$ empirical source error on $n_y$ source examples. Let $I_y$ be a class-wise conditional mutual information term, defined as the information that per-example loss differences reveal about which examples of class $y$ were in the training half of a doubled sample. Let $D_y$ be a class-conditional divergence between source and target feature distributions. We will prove a bound of the form
$$\mathcal{R}_T(y) \le \widehat{\mathcal{R}}_S(y) + \sqrt{\frac{2 I_y}{n_y}} + c\,\sqrt{D_y},$$
where the square-root forms come from the sub-Gaussian change-of-measure arguments used in the supersample and KL-based bounds. Here $c$ is a loss-range constant.
The point of departure is the third term. The domain discriminator, conditioned on a class (soft pseudo-label), is a density-ratio estimator. Its logits give a Donsker–Varadhan lower estimate of $D_y$. The adversary therefore stops being only an opponent and becomes the instrument that measures one term of the bound.
Two claims follow, and both can be tested.
First, the per-class bound ranks per-class target errors better than holistic quantities do, such as global discriminator accuracy or the global source train–validation gap.
Second, the adversarial gradient itself raises $I_y$ for classes that resist alignment. The adversary wins on those classes by memorizing source idiosyncrasies. Reallocating adversarial strength across classes according to the bound's sensitivities should therefore beat both uniform adversaries and global memorization penalties.
Relation to Prior Work
The closest work is Information-Theoretic Analysis of Unsupervised Domain Adaptation. It splits target risk into a population-shift term and an empirical-to-population term controlled by weight–data mutual information. It names a "memorization trap" and derives a gradient penalty (KL-GP). It motivates our memorization term, but it is holistic, and it explicitly leaves the adversarial minimax dynamics unanalyzed. We work at the class level and study the discriminator directly.
Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation derives mutual-information bounds that require noise injection (stochastic gradient Langevin dynamics). It turns gradient-norm terms into weights over source domains. We borrow that move, "derive weights from the bound," but apply it to classes. We also use supersample conditional mutual information, which stays finite for ordinary deterministic SGD; the noise-injection requirement is exactly what the information-density work flags as limiting Generalization Bounds via Information Density and Conditional Information Density.
The class-wise term comes from Class-wise Generalization Error: An Information-Theoretic Analysis. That paper supplies the loss-difference conditional mutual information estimator and the finding that class-wise error varies widely even on balanced data. It assumes training and test come from one distribution; we extend it across a domain shift.
Two papers constrain the design. Fundamental Limits and Tradeoffs in Invariant Representation Learning proves that standard adversarial learning lies strictly inside the achievable accuracy–invariance region. It certifies only static representations, not training trajectories. Our hypothesis about the wasted gap is that it is spent on memorizing source data in hard classes. Maximizing conditional independence for unsupervised domain adaptation already enforces class-conditional alignment, using a kernel surrogate for $I(Z;\text{domain}\mid Y)$. That paper shows class-conditional alignment helps. We therefore must show that the bound-derived weighting, and not class-conditioning itself, is what adds value.
KL Guided Domain Adaptation bounds target risk by a reverse KL divergence on marginal features. It informs our choice of divergence direction for $D_y$. Its marginal-only alignment is the weakness that class conditioning addresses.
Research Program
Theory. We first prove the per-class bound under bounded loss. The novel step is coupling the supersample construction on the source with a change of measure from source to target features within each class. Pseudo-label error adds a term that we will bound by the target classifier's own class-confusion rate. We will report that term separately rather than hide it.
Diagnostic study. Next we estimate every term for ERM, DANN, KL-GP and MCI. The expensive estimator draws many supersamples: roughly twenty random source halvings per method on Digits and Office-31. We compute $I_y$ from loss differences and $D_y$ from a held-out class-conditioned discriminator. We then test whether $I_y$ grows under adversarial training specifically in the classes with high $D_y$.
Method. Finally we build the method, which we call bound-weighted adversarial adaptation. Each class gets its own reversal strength $\lambda_y$. It is updated online in proportion to the bound's sensitivity to $D_y$ and damped when a cheap proxy for $I_y$ rises. The proxy is the per-class loss difference between a training split and a small held-out source split, which is the quantity the class-wise bound tracks. We state plainly that the online signal is a proxy. One diagnostic experiment checks how well it correlates with the full supersample estimate.
Baselines
Naive reference: source-only ERM. This is the reference point. Information-Theoretic Analysis of Unsupervised Domain Adaptation reports 65.9% on SVHN→MNIST, and Domain-Adversarial Training of Neural Networks reports 52.25% on MNIST→MNIST-M.
DANN. This is the adversarial method we modify, so it is the most direct comparison. It is reported at 76.66% on MNIST→MNIST-M and 73.0% on Office Amazon→Webcam Domain-Adversarial Training of Neural Networks. It is reported at 57.7% on VisDA17 KL Guided Domain Adaptation.
KL-GP: the strongest existing information-theoretic competitor. It already penalizes memorization, holistically. It is reported at 71.9% on VisDA17, 93.8% on SVHN→MNIST, and 88.0% on 60° RotatedMNIST Information-Theoretic Analysis of Unsupervised Domain Adaptation.
MCI: the strongest class-conditional competitor. It is reported at 90.9% average on Image-CLEF Maximizing conditional independence for unsupervised domain adaptation. For Office-Home, no reported result was found on Lacuna for MCI.
Fair-comparison protocol. All methods are rerun in one codebase with the same backbone, data splits and epochs. Each gets an equal hyperparameter search budget of the same number of trials. Selection uses a source-only validation criterion, since tuning on target labels would favor whichever method has more knobs. Published numbers serve only as a sanity check that our reruns are close.
Ablations that isolate the key ingredient. The first ablation is a class-conditioned adversary with uniform $\lambda$. It has the same architecture and pseudo-labels, with no bound weighting. The second weights classes by $D_y$ alone, dropping the memorization term. The third is DANN plus a global gradient penalty. Beating means a mean target-accuracy gain over the best of KL-GP, MCI and DANN+GP that exceeds seed variance across five seeds. It must hold on at least three of four benchmarks, with the largest gains on the worst-performing target classes.
Why not just…?
Why not just add KL-GP's gradient penalty to DANN? This is the most likely objection, and it is in our ablations. A global penalty shrinks memorization equally in classes that are already aligned, which costs source fit for no target benefit. The class-wise paper's fourfold spread in per-class generalization Class-wise Generalization Error: An Information-Theoretic Analysis suggests a uniform penalty is badly allocated. If DANN+GP matches our method, the class-level claim fails, and we will report that.
Why not just use a class-conditional alignment method such as MCI? MCI equalizes alignment pressure across classes and ignores memorization entirely. The uniform class-conditioned adversary ablation separates "conditioning" from "bound weighting." If the two tie, our contribution reduces to a diagnostic result.
Why not just reweight classes by pseudo-label confidence or by the per-class validation gap? This is a real threat, because our online memorization proxy is a per-class held-out gap. The theory's added value is the specific combination with the discriminator-estimated $D_y$ and the functional form of the weights. We will run confidence-weighted and gap-only weighting as additional ablations. If either matches the full method, we will conclude that the bound explains the heuristic but does not improve on it.
Proposed Experiments
Benchmarks. We use Digits (SVHN→MNIST, MNIST→MNIST-M), RotatedMNIST at 60° and 75°, Office-31, Image-CLEF and VisDA17. Amazon Reviews with class-drop target shift tests whether class weighting helps when label proportions also shift.
Metrics. We report mean and worst-class target accuracy. We also report the Spearman rank correlation between each class's bound value and its true target error, compared against global discriminator accuracy and the global gap.
Decisive checks. The first check is diagnostic: across classes, DANN's per-class $I_y$ must exceed ERM's more in high-$D_y$ classes than in low ones. Without this, the mechanism is wrong. The second check is predictive: the per-class bound must rank target class errors better than the holistic alternatives on at least Digits and Office-31. The third check tests robustness to pseudo-labels: we repeat the weighting with oracle target labels to bound how much pseudo-label noise costs. We would abandon the weighting method if it passes the first two checks but fails the ablations, and publish the class-wise bound and diagnostic as the contribution.
References
- Information-Theoretic Analysis of Unsupervised Domain Adaptation
- Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation
- Generalization Bounds via Information Density and Conditional Information Density
- Class-wise Generalization Error: an Information-Theoretic analysis
- Fundamental Limits and Tradeoffs in Invariant Representation Learning
- Maximizing Conditional Independence for Unsupervised Domain Adaptation
- KL Guided Domain Adaptation
- Domain-Adversarial Training of Neural Networks