Many fairness methods train a network so that its internal representation carries no trace of a protected attribute such as gender or race. The usual technique, adversarial debiasing, pits the model against a small "adversary" network that tries to read the attribute from the representation, and pushes the model to defeat it. Beating a small in-training adversary, however, does not mean the model has stopped using the attribute. The model may simply have hidden the attribute in a form that particular adversary cannot read. We propose to test this directly, and from outside the model. The test perturbs an input imperceptibly so that a separate, public attribute classifier sees a different gender or race, without ever touching the task label. It then asks whether this perturbation, carried over to a debiased model the attacker cannot inspect, changes the task decisions of that model.
The second research direction supplies the machinery. Adversarial transferability is the observation that perturbations crafted on one accessible "surrogate" model often fool a different, unseen "target" model trained on the same task. That makes the probe black-box, which is exactly how a deployed model would be attacked. Our claim is that transfer of perturbations along the attribute direction separates debiasing that removes the attribute from debiasing that only hides it. We expect minimax-style debiasing to fall mostly in the second category and input-space counterfactual augmentation mostly in the first. The same probe is also a realistic attack: an adversary with only public models can re-open a demographic gap in a system that was certified fair on clean data.
Core Claim
Let $a_P$ be a public attribute classifier (for example, predicting perceived gender on faces) and $f_T$ be a black-box target task model. An attribute-direction perturbation for input $x$ with inferred attribute $s$ solves
$$\delta^\star(x) = \arg\max_{\|\delta\|_\infty \le \epsilon} \; \mathcal{L}\big(a_P(x+\delta),\, s\big),$$
where $\mathcal{L}$ is cross-entropy and $\epsilon$ is a small pixel budget. This objective only pushes the perceived attribute away from its true value, and never references the task label or any task model. We measure the attribute-conditioned flip rate $F_T(\epsilon)$, the fraction of inputs whose target decision $f_T(x)$ differs from $f_T(x+\delta^\star(x))$. We also measure the change in the demographic parity and equalized-odds gaps after the perturbation is applied.
If $f_T$ truly ignores the attribute, $F_T$ should be no higher than for a random perturbation of the same norm. Our hypothesis has three parts. First, targets debiased by gradient-reversal minimax training will show $F_T$ close to that of an undebiased target, even when their clean fairness gaps are small. Second, targets trained on attribute-flipped adversarial examples will show much lower $F_T$. Third, the gap between the two will track how much attribute information a strong nonlinear probe can still recover from the target's representation, a quantity an outside auditor cannot normally measure.
Relation to Prior Work
The motivating evidence comes from the debiasing literature's own admissions. Standard gradient-reversal debiasing uses a shallow adversary on the representation, as in Demoting Racial Bias in Hate Speech Detection, which trains an MLP dialect adversary against a toxicity encoder. That paper also shows the adversary cannot decouple dialect from toxicity when 97% of dialect examples in DWMW17 are labeled toxic. Learning Adversarially Fair and Transferable Representations (LAFTR) adds a decoder to keep the representation useful. The research file notes that such reconstruction pressure tends to reintroduce attribute proxies. Both papers evaluate fairness on clean inputs only. We keep their training recipes and ask what happens to their fairness under a small, targeted input change.
The closest methodological neighbor is Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasing (AEDA). AEDA crafts essentially our perturbation, an input change that flips a gender classifier while keeping task features, and uses it to augment training data. We use the same perturbation as a test-time probe transferred to other models, and AEDA-trained targets serve as the positive control that should resist it. Fairness-aware Adversarial Perturbation Towards Bias Mitigation for Deployed Deep Models (FAAP) runs in the opposite direction. It learns input noise that blinds a surrogate's latent features to gender and transfers that noise to commercial APIs to reduce bias. FAAP needs white-box latent access to the surrogate and treats transfer as a remedy. We need only an input-output attribute classifier and treat transfer as a threat and a diagnostic. We also test whether FAAP-patched deployments are themselves re-biasable.
Two papers constrain our expectations. Fairness Increases Adversarial Vulnerability proves for linear models that fairness constraints move decision boundaries closer to minority-group points, and proposes Ramp loss as a fix. Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning shows that Black and Female subgroups in UTKFace sit closer to the boundary under white-box attacks. Both findings predict that any perturbation hurts debiased models and minority groups more. That is why our design must separate attribute-directed damage from generic boundary fragility, using same-norm random and untargeted controls. Both papers are white-box. Neither asks whether the damage transfers, or whether it is carried specifically by the protected attribute.
On the attack side, Adversarial Attacks on Fairness of Graph Neural Networks (G-FairAttack) shows that edge flips can widen fairness gaps across several fair graph models, including an adversarial one, with little accuracy change. It works gray-box on discrete graphs. We move the question to continuous images, zero-query transfer, and a perturbation that never optimizes a fairness metric directly. The transfer mechanics follow three papers. Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning shows that models sharing backbones share gradients. Publishing Efficient On-device Models Increases Adversarial Vulnerability shows that sibling models are near-ideal surrogates. Adversarial Vulnerability from Interference Between Features in Superposition argues that shared packed features carry perturbations across architectures. That last argument is our mechanistic bet: if gender is still packed into the debiased representation, an attribute direction learned elsewhere should land on it. Reliable Evaluation of Adversarial Transferability and An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability warn that weak single-surrogate attacks overstate defenses. We therefore always report ensemble-surrogate results.
Research Program
We will build a zoo of CelebA targets for the Attractive and Smiling tasks, with gender as the protected attribute, and UTKFace targets for gender or age tasks, with race as the protected attribute. Each target is trained on one data split. Surrogates and the public attribute classifier are trained on a disjoint split, so no weights or training examples are shared.
The target families are chosen to span debiasing mechanisms. They include a vanilla ResNet; a gradient-reversal model with a single MLP adversary; a variant with adversaries at several intermediate layers, following the Boyu Wang et al. design described in the file; a LAFTR-style encoder-classifier-adversary-decoder; an AEDA_robust model; a Ramp-loss fair model; and a vanilla model wrapped with FAAP noise. Every family is instantiated in ResNet and VGG architectures, with debiasing strength swept, so that we can compare models at matched clean equalized-odds gaps rather than at arbitrary settings.
The attacker side has two tiers. The first uses attribute-direction perturbations crafted on the public attribute classifier alone, or on an ensemble of attribute classifiers. The second uses a vanilla task surrogate for the untargeted controls. For each target we record the attribute-conditioned flip rate, the change in the demographic parity and equalized-odds gaps, overall accuracy, and the input-gradient cosine between the attribute classifier and the target's task score. The cosine is computed offline as an analysis-only quantity.
For each target we also train a strong nonlinear probe on its frozen representation to predict the attribute. The central analysis asks how well the black-box flip rate predicts probe accuracy across the zoo, and whether it predicts better than the clean fairness gap does. A final strand tests a defense that follows from the diagnosis: adding attribute-direction transfer examples, crafted on held-out attribute classifiers, to gradient-reversal training. The question is whether this closes the flip-rate gap to AEDA.
Baselines
The naive reference points are the clean target and the target under uniform random sign noise at the same $\epsilon$. Any flip rate or fairness-gap increase must clear the random-noise level, or the effect is only generic fragility of the kind Fairness Increases Adversarial Vulnerability already predicts.
The strongest competing explanation is ordinary untargeted transfer attack. We run it with an ensemble of vanilla task surrogates, following An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability, at the same $\epsilon$, iteration count, and surrogate compute as our attribute attack. Beating it on the diagnostic question means two things. The attribute-direction perturbation should produce a larger increase in the fairness gap per unit of accuracy lost. Its flip rate should also rank targets by probe-measured attribute leakage better than the untargeted attack success rate does. For pure damage, we expect a group-targeted misclassification attack to be at least as destructive, and we report it too. Its role is to calibrate damage, not to answer the removal-versus-hiding question.
The debiasing methods themselves are baselines for the defense strand. For fairness on clean CelebA Attractive, FAAP reports demographic parity improving from 0.5023 to 0.2704 and difference in equalized odds from 0.5683 to 0.0621, with accuracy cost typically under 1–2% (FAAP). AEDA_robust reports 74.30% balanced accuracy with a model bias score of 3.27 on CelebA (AEDA). We will reproduce each method until its clean numbers are within reach of these before perturbing it. No reported result found on Lacuna for fairness under transferable attribute-direction perturbations in any continuous domain, so that number is ours to establish.
Two ablations isolate the key ingredient. In the first, the perturbation is crafted on the attribute classifier but projected to be orthogonal to the attribute gradient. This keeps the norm and input statistics while removing the attribute push. In the second, the same attribute perturbation is crafted on an attribute classifier that shares the target's backbone. This separates attribute-driven transfer from lineage-driven transfer, the effect documented in Publishing Efficient On-device Models Increases Adversarial Vulnerability.
Why not just…?
Why not just attack one group with a standard transfer attack? Perturbing only minority-group positives with an ordinary misclassification loss will certainly widen the equalized-odds gap, probably more than our perturbation does. That attack measures how easily one can damage a group, not whether the model relies on the attribute. It would damage a perfectly attribute-blind model just as well. We include it as the damage ceiling. Our claim stands only if attribute-direction flips differ systematically between gradient-reversal and AEDA models at matched fragility to this group-targeted attack.
Why not just probe the representation with a stronger classifier? This is the most serious alternative, and in white-box audits it may be sufficient. The asymmetry between shallow in-training adversaries and expressive probes is already flagged as an open problem in the file. Our probe matters because regulators, users, and attackers usually lack representation access, and because probe accuracy does not say whether the recoverable attribute actually drives decisions. If flip rate adds nothing beyond probe accuracy in predicting fairness-gap increase, we will report that the black-box test is a useful stand-in for probing and nothing more.
Why not just retrain with Ramp loss or standard adversarial training? Ramp loss restores boundary distance on clean-data fairness models (Fairness Increases Adversarial Vulnerability). If it also suppresses attribute-direction flips, then the phenomenon is boundary proximity, not hidden attributes. The Ramp-loss target in our zoo tests this directly. A low flip rate for Ramp loss combined with high probe leakage would refute our mechanism. Robustness disparity also appears in the very first epoch and is inherited from pretrained backbones (Fairness Through Robustness), so we compare against randomly initialized and pretrained versions of each method.
Decisive Checks
The hypothesis fails if, at matched clean equalized-odds gaps, gradient-reversal targets show flip rates no higher than AEDA targets. It also fails if neither family's flip rate beats the random and orthogonal-projection controls. The diagnostic claim fails if the Spearman correlation between flip rate and nonlinear-probe accuracy across the zoo is no stronger than the correlation between clean fairness gap and probe accuracy.
The threat claim is supported if ensemble-crafted attribute perturbations at $\epsilon = 4/255$ raise the equalized-odds gap of a FAAP-patched or gradient-reversal CelebA target back toward its undebiased value, while accuracy drops less than under the untargeted attack at the same budget. Finally, if attribute-direction augmentation added to gradient-reversal training brings its flip rate to AEDA's level without losing clean fairness, we will have both a diagnosis and a practical repair.
References
- Demoting Racial Bias in Hate Speech Detection
- Learning Adversarially Fair and Transferable Representations
- Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasing
- Fairness-aware Adversarial Perturbation Towards Bias Mitigation for Deployed Deep Models
- Fairness Increases Adversarial Vulnerability
- Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning
- Adversarial Attacks on Fairness of Graph Neural Networks
- Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning
- Publishing Efficient On-device Models Increases Adversarial Vulnerability
- Adversarial Vulnerability from Interference Between Features in Superposition
- Reliable Evaluation of Adversarial Transferability
- An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability