Debiasing the Unlabeled-Target Loss in Semi-Supervised Domain Adaptation

Semi-supervised domain adaptation trains a classifier from three sources: many labeled images from one domain (say, cartoons), a handful of labeled images from the domain we actually care about (say, photos, often one to three per class), and a large pool of unlabeled photos. Methods in this setting add a label-free loss on the unlabeled photos, such as an entropy penalty that pushes predictions to be confident, a pseudo-label loss, or an adversarial term. That loss can make the final model worse than simply training on all the labeled images, and nothing in current methods rules this out. Safe semi-supervised learning has a clean fix for the single-domain case. One version, called DeSSL, subtracts the same label-free loss computed on the labeled images, so the added term averages to zero and can no longer pull the solution away from what the labels support. That fix assumes the labeled and unlabeled images come from the same distribution. Across domains, that assumption fails.

This note proposes a debiased label-free loss for semi-supervised domain adaptation. Its correction term draws on two groups of labeled images: the few labeled target images, which are unbiased but noisy, and the many labeled source images, reweighted toward the target distribution by a domain classifier. The two corrections are mixed with a coefficient chosen to minimize variance. If the domain classifier's weights are poor, the mixture falls back to the target-only correction, so the method keeps its guarantee of never doing worse than training on the labels alone. If the weights are good, the source images lower the noise of the correction when there is only one labeled target image per class. The domain classifier that adversarial adaptation already trains becomes the tool that estimates this reweighting, rather than only a force for aligning features.

Core Claim

In semi-supervised domain adaptation, the unlabeled-target losses used by current methods can be made asymptotically non-inferior to the "Source + Target" supervised baseline without giving up most of their gains. Two claims are specific and testable. First, the correction built from both source and target labeled images gives lower-variance gradients, and higher accuracy at 1-shot, than a correction built from target shots alone. Second, the debiased method never falls below the supervised baseline, even when the unlabeled target pool is contaminated with unseen classes. Undebiased entropy methods do fall below the baseline in that case.

Relation to Prior Work

Two papers motivate the problem. Semi-supervised Domain Adaptation via Minimax Entropy (MME) plays an adversarial game over prediction entropy on unlabeled target data. It reports that standard adversarial feature alignment often fails to beat supervised training on source plus target labels, and that MME itself is sensitive to its entropy weight. That is direct evidence that label-free target losses can hurt. Not All Parameters Should Be Treated Equally (SPL) shows that unseen classes in the unlabeled pool push deep semi-supervised models below the supervised floor, and that confidence filtering does not stop this. We borrow its contamination protocol.

One paper supplies the method. Don't fear the unlabelled: safe semi-supervised learning via debiasing builds the control-variate correction. It proves consistency and asymptotic non-inferiority when labels are missing completely at random, and it reports that the correction stops pseudo-labeling from degrading MedMNIST Retina. We extend its estimator to a setting where most labeled data comes from a different distribution. The reweighting weights come from a domain classifier of the kind trained in Domain-Adversarial Training of Neural Networks.

Three papers constrain the design. Target contrastive pessimistic risk for robust domain adaptation (TCP) gives a guarantee of not doing worse than the source-only model. It does this by minimizing risk against a worst-case labeling of the target data, but only for least-squares and discriminant-analysis models. We aim at the same kind of guarantee for deep networks, but through unbiasedness rather than worst-case optimization, which does not scale to deep models. Safe Self-Refinement for Transformer-based Domain Adaptation (SSRT) detects collapse by watching how many distinct classes the model predicts in a batch, then rolls the model back to a saved snapshot. That is a heuristic safety net with no bound attached; we test whether it becomes unnecessary once the loss is debiased. Finally, Transferability vs. Discriminability (BSP) shows that adversarial alignment distorts the feature spectrum. That warns us that a domain classifier trained on aligned features will be nearly blind to the domain difference, which shapes where we put our density-ratio estimator.

Research Program

Let $H(x,\theta)$ be any label-free loss on an input $x$ under parameters $\theta$, for example prediction entropy or a pseudo-label loss. Let $U_T$ be the unlabeled target set, $L_T$ the labeled target shots, and $L_S$ the labeled source set. Let $R_{\text{lab}}(\theta)$ be the supervised cross-entropy on $L_S \cup L_T$; this is the "Source + Target" objective and serves as the floor. The proposed objective is

$$\mathcal{L}(\theta)=R_{\text{lab}}(\theta)+\lambda\Big[\overline{H}_{U_T}(\theta)-\alpha\,\overline{H}_{L_T}(\theta)-(1-\alpha)\,\frac{\sum_{x\in L_S} w(x)H(x,\theta)}{\sum_{x\in L_S} w(x)}\Big].$$

Here $\overline{H}_{A}$ is the mean of $H$ over set $A$, and $\lambda$ is the loss weight. The weight $w(x)=\hat p(\text{target}\mid x)/\hat p(\text{source}\mid x)$ is an estimated density ratio from a domain classifier. The coefficient $\alpha\in[0,1]$ mixes the two corrections. When the density ratio is correct, the bracketed term has zero expectation for any $\alpha$, so $\lambda$ can no longer move the solution away from the minimizer of $R_{\text{lab}}$. The unlabeled data then acts only as variance reduction, which is the DeSSL argument carried to two domains.

The coefficient $\alpha$ is set online to the value that minimizes the estimated variance of the bracket. This is the standard optimal-combination rule for two unbiased estimators, and a biased one is caught by the check below. When $\alpha=1$ the method reduces exactly to target-only DeSSL. To protect against a wrong density ratio, we also compare the weighted source mean of $H$ against the target-shot mean. If they differ by more than their joint standard error, $\alpha$ is forced toward 1.

The domain classifier for $w$ is trained on features from the frozen pretrained backbone, not on the features being adapted. The reason is that adversarial alignment deliberately erases domain information from the adapted features, so a classifier on them would report weights near 1 whatever the true shift. We analyze this circularity directly by comparing weights estimated on frozen features with weights estimated on adapted features.

The debiasing wraps existing losses rather than replacing them. We apply it to the entropy term of MME and to the entropy-weighted term of CDAN+E from Conditional Adversarial Domain Adaptation. The feature-alignment game in CDAN compares two domains rather than scoring individual samples, so it is left undebiased, and we report its separate contribution.

On the theory side, we will state consistency and asymptotic non-inferiority relative to the $R_{\text{lab}}$ minimizer when $w$ is correct. We will also derive a bias bound that grows with the error in $w$ and shrinks with $\alpha$.

Baselines

The naive reference point is "Source + Target": supervised training on $L_S \cup L_T$ with the same backbone and schedule, and no unlabeled loss. Every method must be judged against it on a per-task, per-seed basis. MME reports beating this baseline by "nearly 10%" on DomainNet. The file gives no absolute Source + Target number, so we will reproduce it ourselves.

The strongest existing competitor is MME. The file reports it at 66.4% (1-shot) and 68.9% (3-shot) accuracy on DomainNet with a ResNet backbone, in Semi-supervised Domain Adaptation via Minimax Entropy. No reported Office-Home SSDA number was found on Lacuna for MME. MME is the right opponent because it is the method whose entropy term we debias, so the comparison isolates the effect of debiasing. We run it from the same code, backbone, augmentations, and number of training iterations. Its $\lambda$ gets the same tuning budget as ours, and we report both the result at its tuned $\lambda$ and its worst result across the $\lambda$ sweep. Beating MME means matching its mean accuracy within one standard deviation on the clean splits. It also means having zero splits below Source + Target where MME has some, and a higher worst case across the $\lambda$ sweep.

CDAN+E, with and without BSP, is the adversarial-alignment competitor. The file reports CDAN+E at 65.8% and CDAN+BSP at 66.3% on unsupervised Office-Home in Conditional Adversarial Domain Adaptation and Transferability vs. Discriminability. We run it in the SSDA setting by adding the target shots to its supervised loss. SSRT-style rollback, driven by the prediction-diversity collapse detector, is grafted onto undebiased MME to serve as the heuristic-safety competitor.

The key ablation is target-only DeSSL, which is our method with $\alpha$ fixed at 1. Reweighted-source-only, with $\alpha$ fixed at 0, is run as a second ablation, and so is density ratios estimated on adapted rather than frozen features. Together these isolate whether the reweighted source correction, the one new ingredient, earns its place.

Why not just…?

Why not just apply DeSSL using the target shots alone? This is the most serious threat. With 126 classes and 3 shots, $L_T$ holds 378 images, which may be enough for a low-noise correction. Our prediction is that target-only debiasing works at 3-shot but is noisy at 1-shot, especially under imbalanced shots. The $\alpha=1$ ablation settles this. If target-only DeSSL matches our method at 1-shot as well, the reweighting adds nothing, and the paper becomes an empirical study showing that DeSSL alone makes SSDA safe. We would report that honestly.

Why not just hold out some target labels and fall back to Source + Target if adaptation looks worse? With one to three labels per class, a held-out set costs a large fraction of the supervision, and its accuracy estimate is too noisy to separate methods a few points apart. We will run this model-selection baseline with one shot per class held out, and report both its accuracy and how often it picks the wrong model.

Why not just filter unlabeled targets by confidence, or tune $\lambda$ carefully? SPL shows that confidently wrong samples from unseen classes get past confidence thresholds. MME reports sensitivity to $\lambda$, and tuning $\lambda$ needs target labels we do not have. The contamination experiment below tests filtering directly. The $\lambda$ sweep shows whether debiasing flattens that sensitivity.

Why not just add rollback à la SSRT? Rollback reacts after a collapse and says nothing about slow, gradual degradation below the floor. If rollback-equipped MME is never below Source + Target, our safety claim loses its practical force, and only the calibration and variance arguments would remain.

Proposed Experiments

We use the DomainNet 126-class SSDA splits from Moment Matching for Multi-Source Domain Adaptation and Office-Home from Deep Hashing Network for Unsupervised Domain Adaptation, at 1-shot and 3-shot, with five seeds per task. Metrics are mean target accuracy, the number of (task, seed) pairs below Source + Target, the worst-case accuracy gap to Source + Target, Expected Calibration Error, and worst-class accuracy.

Three stress tests are designed to trigger degradation. In the contamination test, a fraction (0–60%) of the unlabeled target pool is replaced with images from held-out DomainNet classes, following the SPL mismatch protocol. In the imbalance test, shots are drawn with a long-tailed per-class count. In the severe-shift test, the domain pairs involving Quickdraw are used, where the SSRT paper reports collapse to 0.3% accuracy under standard training.

Two diagnostics accompany the accuracy results. We measure the variance of the correction term's gradient under $\alpha=1$, $\alpha=0$, and the learned $\alpha$. We also plot the learned $\alpha$ against the domain classifier's calibration, to check that the fallback triggers when the weights are wrong.

The decisive checks are these. On contaminated DomainNet, debiased MME must stay at or above Source + Target at every contamination level while undebiased MME drops below it. At 1-shot, the learned-$\alpha$ variant must beat $\alpha=1$ by a margin larger than the seed-to-seed spread. On clean splits, accuracy must stay within about one point of MME's reported 66.4% (1-shot) and 68.9% (3-shot). If the first check fails, the safety claim is false. If only the second fails, the contribution shrinks to applying DeSSL to domain adaptation.