Fair Transfer Without a Moving Target: Replacing the Cascaded Adversaries in Cross-Domain Debiasing

Sometimes a model has to be made fair in a setting where the sensitive attribute, such as sex or race, is recorded in the training data but missing in the deployment data. The best-known method for this case chains two adversarial games. In the first, a network learns features that hide which dataset an example came from, so that a sensitive-attribute predictor trained on the source data can be used on the target data. In the second, another network tries to recover those inferred attributes from the task model's internal features, and the task model learns to defeat it. Each game on its own is known to oscillate, collapse, and depend heavily on learning rates. Chained together, the second game chases labels that the unstable first game keeps changing. When training finishes, nobody can tell whether the features are actually fair or whether the adversary simply stopped learning.

We propose to break the chain. First, the attribute estimator is trained and then frozen, so its outputs no longer change during training. Second, the fairness adversary is replaced by a closed-form discrepancy that needs no inner optimization. This is a kernel distance between the feature distributions of the two groups, where each target example is weighted by the estimator's predicted probability of belonging to each group rather than by a hard guessed label. We also make instability measurable. After training, we fit a fresh probe to the frozen features and report how much sensitive information it recovers compared with what the training adversary claimed. We call this difference the audit gap. The central question is whether this setup gives target-domain demographic parity that is as good or better, with much lower variance across seeds and hyperparameters, than dual-adversarial training. We also ask whether the benefit comes from removing the fairness adversary, from freezing the estimator, or from neither.

Core Claim

The closest existing work is FAIRDA (Fair classification via domain adaptation: A dual adversarial learning approach; arXiv version Joint Adversarial Learning for Cross-domain Fair Classification). FAIRDA couples a domain discriminator with a bias-predicting adversary. It proves that parity holds in the target domain as long as the inferred attributes are "not completely random." On CelebA it reports a 21.2% improvement in demographic parity over a vanilla model, with a 0.1% drop in accuracy. FAIRDA motivates this proposal. Our claim is that most of its fragility comes from the second loop. That loop is a minimax game whose targets are noisy, and noisy in a way that changes over time. We think a non-adversarial penalty on posterior-weighted group distributions can remove this fragility while keeping FAIRDA's guarantee.

The weighted penalty has a simple justification. Let $q(s\mid x)$ be the frozen estimator's probability that example $x$ belongs to group $s \in \{0,1\}$. Let $z = h(x)$ be the task model's features. The penalty is

$$ \mathcal{R}(h) = \mathrm{MMD}_k^2\Big(\textstyle\sum_i w^1_i \delta_{z_i},\; \sum_i w^0_i \delta_{z_i}\Big), \qquad w^s_i = \frac{q(s\mid x_i)}{\sum_j q(s\mid x_j)}. $$

Here MMD, the maximum mean discrepancy, measures the distance between two distributions through their average embeddings under a kernel $k$. The symbol $\delta_{z_i}$ denotes a point mass at $z_i$, and the $w^s_i$ are per-example weights for group $s$ computed over target examples.

Under the usual mutual-contamination assumption that estimator errors are independent of $z$ given the true group, each weighted distribution is a known mixture of the two true group distributions. The penalty then equals the true group discrepancy multiplied by $(1-\epsilon_0-\epsilon_1)^2$, where $\epsilon_s$ is the estimator's confusion rate for group $s$. This is a quantitative version of FAIRDA's "not completely random" condition. It also says how much to scale the penalty: $\epsilon_s$ can be estimated on held-out labeled source data.

The independence assumption is likely false, because the estimator and the encoder see the same inputs. Testing how badly it fails is part of the work.

Research Program

The work is a factorial study built around one new method.

The two components that vary. The domain-alignment component is either an adversarial discriminator trained through gradient reversal, which flips the sign of gradients passed back to the encoder, or a closed-form MMD alignment. The fairness component is either FAIRDA's bias adversary or the weighted MMD penalty above. We also cross each combination with whether the attribute estimator is jointly trained or frozen after stage one.

Why keep adversarial alignment in the grid. The closed-form discrepancies in the stabilization literature were tested mostly on tabular data. These include the parametric sigmoid metric of Learning fair representation with a parametric integral probability metric and the kernel metrics of Fair Representation Learning for Continuous Sensitive Attributes using Expectation of Integral Probability Metrics and Fair Representation Learning with Maximum Mean Discrepancy Distance Constraint. There is also a negative signal for plain MMD on images. Towards Fair Knowledge Transfer for Imbalanced Domain Adaptation reports DAN, an MMD-based adaptation method, at 51.1% on imbalanced Office-Home, against 65.6% for its own method. We therefore do not assume that replacing the domain adversary is a free improvement. The grid is designed to show which loop can safely be made non-adversarial.

How the fairness penalty is applied. The penalty is computed on the scalar task output as well as on $z$. Computing it on the one-dimensional output avoids the dimensionality problem that A Fair Classifier Using Kernel Density Estimation reports for density-based fairness penalties.

Measuring instability. We measure it in three ways, beyond reporting mean performance. The first is the spread of target demographic parity and accuracy across ten seeds. The second is hyperparameter sensitivity: the fraction of a fixed grid of learning rates and penalty weights that lands within a tolerance of the best fairness-accuracy trade-off. The third is the audit gap: after training, a fresh multilayer perceptron is trained on frozen target features to predict the true held-out sensitive attribute, and we compare its accuracy with the training adversary's final accuracy.

The audit gap addresses the problem raised in Learning Controllable Fair Representations, which shows that fairness guarantees degrade when the adversary is undertrained. It also addresses the learning-rate-driven oscillations documented in Fairness-Aware Neural Rényi Minimization for Continuous Features.

Continuous attributes (secondary extension). A secondary step extends the weighting to continuous attributes such as age. The method would weight by the estimator's predictive density and apply the per-value kernel smoothing of the expectation-of-IPM approach cited above. No method in the file handles continuous attributes that are unobserved in the target domain.

Baselines

Vanilla source-only classifier. The naive reference is a task model trained on source data without adaptation or debiasing, then applied to the target. FAIRDA uses this as its reference point. It is trained with the same backbone and the same epochs as every other method. It sets the floor for target demographic parity and the ceiling for "no accuracy cost."

FAIRDA. This is the strongest existing method for the exact problem. Its reported figure is a 21.2% improvement in demographic parity over vanilla on CelebA with a 0.1% accuracy drop (Fair classification via domain adaptation). The file gives no absolute FAIRDA numbers for COMPAS, Adult, or Toxicity. We will reimplement FAIRDA on its own source/target splits and first reproduce the relative CelebA gain. Every method then gets the same search budget: a fixed grid of learning-rate ratios and penalty weights, ten seeds per configuration, with model selection on a small labeled target validation set that has the sensitive attribute revealed. Beating FAIRDA means one of two outcomes. Either equal or better mean target demographic parity and equal opportunity at matched accuracy, with a clearly smaller seed spread and a smaller audit gap; or a better fairness-accuracy trade-off across the whole grid, not just at the best configuration.

FAIRDA with Environment Label Smoothing on both discriminators. This baseline applies the fix from Free Lunch for Domain Adversarial Training: Environment Label Smoothing. That work softens discriminator targets so gradients do not vanish when the discriminator becomes overconfident. It raised Rotating MNIST accuracy from 52.1% to 62.1% and reached 84.6% on Office-Home with a ViT backbone. We apply smoothing to both FAIRDA discriminators, with the smoothing strength added to the tuning grid. This is the most serious competitor to our stability claim, because it is a one-line change.

Ablations isolating the key ingredients. Two ablations separate the contributions. The first is FAIRDA with a frozen estimator, keeping both adversaries, which isolates the effect of breaking the cascade. The second is our penalty with hard argmax labels instead of posterior weights, which isolates the weighting.

Why not just…?

Why not just drop the sensitive attribute? In the target domain the attribute is already absent, so unawareness is exactly what the vanilla model does, and FAIRDA reports it as severely unfair on CelebA. Fairness-Aware Neural Rényi Minimization for Continuous Features and Dynamically Disentangling Social Bias from Task-Oriented Representations with Adversarial Attack show why: other features act as proxies for the attribute. The vanilla baseline covers this alternative directly.

Why not just add label smoothing to FAIRDA's two discriminators? This is a real threat. ELS fixes vanishing gradients, but it does not obviously fix the moving-target problem, in which the bias adversary's labels shift as the domain encoder moves. Our factorial design separates the two causes. Suppose FAIRDA with ELS matches our method on seed variance and audit gap, and the frozen-estimator ablation adds nothing. Then gradient vanishing was the whole story, and we would report ELS as the answer.

Why not just use a linear projection that removes the sensitive direction? On DIAL, Dynamically Disentangling Social Bias reports that iterative null-space projection drops accuracy from 0.73 to 0.62, while the dynamic method keeps 0.73 and cuts the true-positive-rate gap from 0.40 to 0.09. Linear removal also misses the nonlinear dependence that the audit probe is designed to detect. We include it only as a secondary check on CelebA features.

Why not just use the shuffle-based penalty of Fair Supervised Learning with A Simple Random Sampler of Sensitive Attributes? That method reaches a statistical parity of 0.001 on Adult. However, it assumes the attribute is observed and still trains a discriminator. We will run a variant that shuffles posterior-sampled target attributes, as an alternative fairness component within the grid.

Proposed Experiments

Datasets. We use FAIRDA's four domain splits: CelebA with sex as the attribute, Adult, COMPAS, and Toxicity. For a harder vision setting we add the skin-lesion transfer of Achieving Reliable and Fair Skin Lesion Diagnosis via Unsupervised Domain Adaptation, with Fitzpatrick skin type withheld in the target. That work reports 89.4% AUROC on ISIC 2020 with multi-source adaptation and "fairness for free" across skin tones. It therefore also tests whether explicit debiasing adds anything beyond what adaptation already provides.

Metrics. We measure target demographic parity, equal opportunity, and accuracy or AUROC; seed spread; grid sensitivity; and the audit gap.

Decisive checks. Three results would settle the question. The first is whether dual-adversarial training shows a larger audit gap and larger seed spread than our method at matched accuracy, since this is the core empirical claim. The second is whether the measured penalty scales with the estimated $(1-\epsilon_0-\epsilon_1)^2$ on semi-synthetic data where we inject controlled attribute noise. If it does not, the weighting argument fails. The third is where the gain comes from within the factorial grid: from the fairness loop, from freezing the estimator, or, if ELS closes the gap, from neither.