Patches That Discriminate: Group-Selective Physical Attacks and a Cheap Token Defense for Face Attribute Models

Most work on attacking the fairness of machine learning models stays inside the computer. It flips group labels, poisons training images, or adds invisible pixel noise. Most work on physical attacks, such as printed stickers, adversarial glasses, or camouflage that survives a real camera, aims only at causing errors or impersonating someone. This proposal joins the two. We ask whether a single physical artifact can be optimized to make a deployed face-analysis model fail mostly for one demographic group while overall accuracy stays nearly unchanged. Examples are a sticker on a kiosk camera lens, a printed backdrop behind the capture point, or a pair of glasses whose design spreads within a community. An artifact like that would pass any audit that watches average accuracy. It would also be hard to trace, because the system's measured unfairness would look like ordinary demographic bias rather than an attack.

The second half of the proposal is a defense that fits the real constraint: retraining large vision models with adversarial training is expensive. Adversarial training means repeatedly training on worst-case perturbed inputs. Its fairness-aware variants cost even more. We keep a pretrained Vision Transformer frozen and train only a handful of extra input tokens. These tokens are trained against an attacker that maximizes the gap between groups, not against one that maximizes average error. Our bet is that the choice of attacker objective, rather than the capacity of the defense, decides whether cheap defenses protect the groups that need protection most.

Core Claim

We make two claims. The first is that the physical capture path does not wash out fairness attacks. Rendering-aware optimization can produce a group-selective physical artifact. Under it, the Equalized Odds gap of a fair face attribute classifier grows several-fold, while the drop in aggregate accuracy stays below what an accuracy monitor would flag. The Equalized Odds gap is the difference between groups in true-positive and false-positive rates.

The second claim is that a defense of roughly ten learnable tokens on a frozen backbone recovers most of this gap. It does so when trained against a disparity-maximizing, physically rendered attacker, at a small fraction of the cost of full fair adversarial training. Training the same tokens against an accuracy-maximizing attacker does not recover the gap. That last contrast is the key ablation. If it fails, the proposal reduces to "parameter-efficient defenses also help fairness a bit," which is not worth a paper.

The attack design follows a result by Gao et al., On the Alignment between Fairness and Accuracy (companion version). That paper shows that an attack on fairness moves logits in the same direction as an accuracy attack on specific subgroups: true positives of the disadvantaged group and true negatives of the advantaged group. It also shows that breaking Demographic Parity almost always breaks Equalized Odds. A group-selective universal patch is therefore just a universal patch with a subgroup-weighted objective. That framing makes the attack tractable and also tells the defense where to look.

Relation to Prior Work

Three papers motivate the gap. On the Alignment between Fairness and Accuracy shows digital feature perturbations push the Demographic Parity gap on Adult from about 14% to nearly 100%. Its Fair Adversarial Training (FAT) holds the gap near 15–20%, but the authors flag the multi-step training cost as unresolved, and the attacks are only $\ell_\infty$ noise. On Adversarial Robustness of Demographic Fairness in Face Attribute Recognition shows clean-label poisoning raises the CelebA Equalized Odds disparity from 0.0392 to 0.3896. Its full-retraining Robust-Fair Training defense recovers only to 0.2456, and it never considers physical artifacts. Fairness Through Robustness shows that Black and Female subgroups on UTKFace sit closer to the decision boundary even when accuracy is equal, and that this bias is mostly set in the first epoch and inherited from ImageNet pretraining. That is exactly why a frozen pretrained backbone is a credible place for a group-selective attack to land.

Two papers supply our methods. Face3DAdv builds 3D digital twins of faces and optimizes wearable patches through a differentiable renderer, using importance sampling over poses. It reports 94.42% digital and over 64% physical impersonation success on ArcFace. We reuse its rendering pipeline but replace the impersonation objective with a group-disparity objective. Face3DAdv measures no group differences. Robustness Tokens trains ten tokens on a frozen ViT in about 200 steps and recovers DINOv2-Giant from 0.3% to 63.1% accuracy under white-box attack. We reuse its architecture and its feature-correction loss. Its training is fairness-blind, it uses only digital noise, and its robustness partly depends on keeping the tokens secret. We drop that last assumption.

Two lines of work constrain the design. To be Robust or to be Fair and DAFA show that standard adversarial training widens class-wise disparities. They show that fixing this requires non-uniform treatment of hard classes, through reweighting and enlarged per-class margins. This raises the open question our ablation targets: can a frozen backbone plus tokens carve group-specific margins at all? On the physical side, ARMOR shows that random patch injection without any min-max loop can harden detectors. That is a serious cheap competitor, discussed below. PatchCURE certifies patch robustness at near-undefended throughput, but it assumes a known patch size and reports no group-level results.

Research Program

The attack. We consider a face attribute classifier: smiling or attractiveness on CelebA with gender as the protected attribute, and age bracket on FairFace and UTKFace with race as the protected attribute. Two artifact families are studied. The first is worn accessories (glasses frames, a respirator band), rendered onto 3D face twins as in Face3DAdv. The second is capture-path artifacts that every image passes through: a semi-transparent lens sticker, modeled as a blurred, color-limited overlay, and a printed backdrop region. The artifact is universal, meaning one pattern is used for everyone.

Its texture $p$ is optimized as

$$\max_{p}\; \mathbb{E}_{T}\Big[\Delta_{\mathrm{EO}}\big(f(R_T(x,p))\big)\Big] \;-\; \beta\,\Big|\mathrm{Acc}\big(f(R_T(x,p))\big)-\mathrm{Acc}(f(x))\Big|.$$

Here $R_T$ renders the artifact into image $x$ under a sampled physical transformation $T$, which covers pose, lighting, print color gamut, and blur. $\Delta_{\mathrm{EO}}$ is a differentiable surrogate of the Equalized Odds gap, built from the subgroup terms identified by the alignment result. $\mathrm{Acc}$ is aggregate accuracy, and $\beta$ trades disparity against stealth.

We sweep $\beta$ to trace a stealth–disparity frontier. We also test transfer from a surrogate model to the deployed one.

The defense. We freeze a pretrained ViT (DINOv2 or OpenCLIP backbones, as in Robustness Tokens) with a linear attribute head and train only about ten appended tokens. The cost saving comes from two sources. The backbone gets no gradient updates. The attacker is a single universal artifact, which is warm-started and updated for a few steps per batch rather than re-solved per sample with multi-step projected gradient descent.

The training loss combines three terms. The first is the Robustness Tokens clean-invariance term. The second is a feature-correction term on rendered, disparity-maximizing artifacts. The third is a group-weighted classification term whose weights rise for whichever group currently has the worst robust error. This adapts the reweighting idea from Fair-Robust-Learning to tokens.

We report wall-clock GPU-hours throughout. The defense is evaluated against a fully adaptive attacker who knows the token values. Robustness that rests on token secrecy does not count.

Baselines

The naive reference is the undefended frozen backbone with a standard fairness-regularized head. It shows how much disparity the attack produces when nothing is done.

The strongest existing method is full-parameter fair adversarial training: FAT from On the Alignment between Fairness and Accuracy, with its inner loop replaced by our rendered patch attacker so that it faces the same threat. The closest reported figure is digital, not physical: FAT keeps the Adult Demographic Parity gap at about 15–20% against roughly 100% undefended. On face data, the Robust-Fair Training defense of Demographic Fairness in Face Attribute Recognition reaches an Equalized Odds disparity of 0.2456 under poisoning on CelebA attractiveness with LightCNN-9. We found no reported result on Lacuna for any defense against physical, group-selective patch attacks.

All methods share the same data splits, the same attacker family and budget at evaluation, and the same hyperparameter search budget, counted in trials. Full fine-tuning is given its natural larger compute and reported with GPU-hours beside it. "Beating it" means one of two outcomes. Either the token defense comes within about 0.02 of FAT's Equalized Odds gap under adaptive physical attack at under a tenth of its training cost, or it matches FAT's worst-group robust accuracy at that cost.

The fairness-blind efficient baseline is Robustness Tokens as published, trained against accuracy-maximizing digital noise.

The key ablation is the same token defense trained against accuracy-maximizing rendered patches instead of disparity-maximizing ones, with every other setting identical. This isolates the attacker objective.

The class-fair adversarial training methods, Fair-Robust-Learning and DAFA, are run on the face tasks with groups standing in for classes. They show whether margin-based methods built for class disparity transfer to demographic groups. Their reported CIFAR-10 numbers (worst-class robust accuracy 21.31% to 30.53% for DAFA; worst-class robust error 82.7% to 69.5% for FRL) serve only as a sanity check on our reimplementation.

Why not just…?

Why not just inject random patches during training, as ARMOR does? This is the most serious threat. ARMOR gets 0.62 confidence under white-box attack, against 0.14–0.30 for prior defenses, with no min-max loop. Against fairness attacks, however, the alignment paper reports that training on random perturbations without joint fairness terms fails to close demographic gaps. Random patches carry no information about which subgroup the attacker will target. We run random patch injection, with and without the group-weighted loss, on the frozen-token setup at matched cost. If it matches the disparity-trained tokens, we will report that the adversarial inner loop is unnecessary and reframe the contribution around the attack and the audit.

Why not just add a fairness regularizer or adjust per-group thresholds after training? Statically fair models collapse under attack, as the Adult result from about 14% to nearly 100% shows. Thresholds tuned on clean data are exactly what a capture-path artifact shifts. We nonetheless include re-tuning per-group thresholds on artifact-corrupted validation data. This defender sees the attack, which makes it an optimistic baseline. Our attacker is then re-optimized against the re-tuned thresholds.

Why not just deploy a certified patch defense like PatchCURE? PatchCURE needs a known patch size. Translucent lens overlays and backdrops are neither small nor contiguous. Its feature masking may also remove fine facial cues unevenly across groups. We measure its clean and certified accuracy per group on FairFace, using its ViT-SRF variant. This tests whether the defense itself creates disparity.

Proposed Experiments and Decisive Checks

The first check concerns the attack. On CelebA and FairFace we render artifacts through the 3D twin pipeline and report the Equalized Odds gap, the Demographic Parity gap, and worst-group accuracy against aggregate accuracy drop, along the $\beta$ frontier. We then print the two strongest artifacts, a lens sticker and glasses, and capture a small consented volunteer set spanning FairFace's race groups. The test is whether the disparity seen in simulation survives physical capture. If the physical gap falls to the level of a random patch of the same size, the first claim fails.

The second check concerns the defense. We compare the token defense trained on disparity-maximizing artifacts, the token defense trained on accuracy-maximizing artifacts, random patch injection, Robustness Tokens, the threshold baseline, and full fair adversarial training. All are evaluated under a token-aware adaptive attacker, and we report Equalized Odds gap, worst-group robust accuracy, clean accuracy, and GPU-hours. The second claim holds only if two things are true: the disparity-trained tokens beat the accuracy-trained tokens on gap and worst-group accuracy by a margin larger than the seed variance, and they come close to full fair adversarial training at a fraction of the cost.

The third check is diagnostic. On UTKFace we measure per-group boundary distance before and after token training, using the curve metric from Fairness Through Robustness. This tells us whether tokens actually widen margins for the closer groups. If they do not, it shows directly that frozen backbones cannot be remargined, which would answer the open tension between parameter-efficient and margin-based robust fairness in the negative.

Any released attack code will default to the auditing objective, measuring disparity under a given artifact, and will not ship the optimized textures. This is a modest guard against the dual-use risk that fairness-attack work carries.