Suppose a large pretrained vision model, frozen and too costly to retrain, produces face features that a third party will use for some legitimate task, such as recognizing expressions. Those same features also reveal gender, age, and race, and an adversary can perturb input images to break the legitimate task. We propose to address both threats with a handful of extra learnable input tokens appended to the frozen model. The tokens are trained so that the model's output features keep the legitimate task working, stay stable under adversarial input noise, and carry no recoverable information about several sensitive attributes. That includes their combinations, such as "young woman", which can stay guessable even when each attribute alone is hidden. The training cost should stay almost flat as more attributes are protected, because one shared adversarial example and one small joint attribute-guessing network replace the per-attribute machinery used in prior work.
This matters because current multi-attribute privacy methods retrain whole networks, and the only methods that make adversarial robustness cheap ignore privacy. Prior work also gives reason to believe the two goals fight each other inside the representation. The decisive question is whether a few tokens on a frozen backbone can remove sensitive information that is tangled into the same feature channels the model needs for accuracy and robustness. The alternative is that this cheap setting simply recreates the utility collapse seen under full retraining.
Core Claim
The claim has three parts. First, prefix tokens on a frozen Vision Transformer (ViT), an image model that processes images as sequences of patch tokens, can be trained to protect $m$ correlated attributes jointly, robustly, and at roughly constant cost in $m$. Second, protecting the joint attribute configuration closes a leakage channel that per-attribute adversaries leave open. Third, the utility cost of each (target, private) attribute pair is still predicted by how much the two share feature channels, so the frozen-backbone setting inherits rather than escapes the correlation limits seen under full retraining. Concretely, with frozen encoder $f$, learnable tokens $t$, released feature $z = f(x; t)$, utility head $h$, and a joint attribute adversary $g$ that predicts the full vector $\mathbf{a} \in \{0,1\}^m$ of private attributes as one categorical label over its $2^m$ configurations, we train
$$\min_{t,h}\ \mathcal{L}_u\big(h(z), y\big) + \mathcal{L}_u\big(h(z_\delta), y\big) + \lambda\,\big[\mathrm{CE}(\mathcal{U}, g(z)) + \mathrm{CE}(\mathcal{U}, g(z_\delta))\big],\qquad \min_g \mathrm{CE}\big(\mathbf{a}, g(z)\big).$$
Here $y$ is the utility label and $\mathcal{L}_u$ is the utility loss. The perturbed feature is $z_\delta = f(x+\delta; t)$, where $\delta$ comes from a few steps of projected gradient attack on the utility loss. $\mathcal{U}$ is the uniform distribution over attribute configurations, and $\lambda$ trades privacy against utility. The privacy terms reuse the forward passes already paid for by the robustness term. Each step therefore costs about $K+1$ backbone passes for a $K$-step attack, regardless of $m$. The only per-attribute cost is a larger output layer on a small head.
How This Relates to Prior Work
Three lines of multi-attribute privacy work motivate the problem, and all three require full training. PrivacyNet trains a conditional CycleGAN with four subnetworks to selectively hide age, gender, and race in images while preserving face matching. EIGAN/D-EIGAN runs a minimax game between an encoder, $n$ utility networks, and $m$ attribute adversaries. It contributes the useful finding that pushing adversary outputs toward uniform with cross-entropy is more stable than divergence objectives, and we reuse that technique. FlowSAN stacks autoencoders to fool unseen gender classifiers, which multiplies inference cost and degrades matching beyond three stages. None of the three defends against adversarial inputs, and all of them pay gradient cost across every network at every step. Privacy-aware recommendation with adversarial learning supplies the argument that protecting several traits together beats protecting one, because it denies the encoder shortcuts. Our joint adversary takes that argument one step further, to combinations of attributes.
Two papers constrain the design. ARPRL combines robustness and privacy with four networks and mutual-information objectives. It reports that robust-only training maximizes attribute leakage and that naively stacking separate robust and private defenses compounds clean-accuracy loss. It also proves that privacy imposes a non-zero floor on utility error. However, it handles one private attribute and offers no cost reduction. Zhang et al. on attribute correlation shows that when a target and a private attribute share feature channels (high "feature reliability"), scrubbing the private one destroys the target, with correlation up to 0.68 between that score and the achieved trade-off. That work is limited to single attribute pairs under full retraining. We test whether its prediction survives in the multi-attribute, frozen-backbone regime.
Two papers supply the efficiency mechanism, and both ignore privacy. Robustness Tokens freezes ViTs such as DINOv2 and trains 10 prefix tokens in about 200 steps. It finds that the tokens restore "massive activations", meaning outlier channels that attacks suppress, and notes that its security depends on keeping the tokens secret. FullLoRA-AT shows that plain low-rank adapters fail at adversarial fine-tuning unless LayerNorm parameters are also adapted. With that change it tunes about 5% of parameters in about 75% of the full training time. Our contribution is to give the token mechanism a privacy objective and to measure the tension between the two goals. That tension is a real risk: robustness may depend on the very outlier channels that carry demographic information.
Research Program
We build the token defense on frozen DINOv2 backbones, the setting where Robustness Tokens was validated. The private attributes come from CelebA (Male, Young, Pale_Skin, Eyeglasses, scaled up to $m=6$) and UTK-Face (gender, race, and age bins). The utility targets are the CelebA attributes used by Zhang et al. (Smiling, Wavy_Hair, Heavy_Makeup) and linear-probe identity retrieval on held-out CelebA identities, with LFW as an unseen evaluation set.
The program has three parts. The first is the method and its cost curve: wall-clock time and memory as $m$ grows from 1 to 6, compared with a multi-adversary design in the style of EIGAN. The second is a mechanism study. For each layer we locate where attribute information is linearly and nonlinearly decodable, and we check whether it concentrates in the massive-activation channels. We then ask whether the tokens hide attributes by suppressing those channels, which should cost robustness, or by redistributing information within them. The third is a correlation study that extends feature reliability to sets of private attributes. We measure each target attribute's feature-channel overlap with the union of private attributes and test whether this score predicts the clean-accuracy loss per target.
The threat model is as follows. The recipient of the released features trains its own attribute probes after the defense is frozen. A separate attacker perturbs inputs to break utility. In a stress-test setting, that attacker also knows the tokens.
Baselines
The naive reference point is the frozen DINOv2 with no tokens, which gives the leakage level, clean accuracy, and robustness collapse we start from. For ImageNet, Robustness Tokens reports that DINOv2-Giant falls to 0.3% accuracy under white-box attack and recovers to 63.1% with tokens, with clean accuracy of 85.2% versus 85.6%. We expect a similar collapse on face attributes. That is a prediction, not a reported number.
The strongest robustness-only competitor is Robustness Tokens trained exactly as published, on our utility heads, with the same token count, number of attack steps, and step budget. Our method beats it only if it reaches matched robust and clean utility while also lowering attribute leakage.
The strongest privacy competitor is a full-fine-tuning, multi-adversary minimax in the EIGAN style, with ARPRL's robustness term added. We give it the same data and the same tuning budget, with matched sweeps over $\lambda$, and we report its compute separately. For leakage, ARPRL reports CelebA attribute inference dropping from 81% to 57% for a single attribute. For frozen-backbone, multi-attribute privacy on CelebA we found no reported result on Lacuna. Beating this competitor therefore means matching its privacy–utility curve within a few points at a small fraction of its GPU-hours, not exceeding it outright.
The higher-capacity efficient competitor is FullLoRA-AT adapters trained with the same joint privacy objective. This separates "tokens" from "any parameter-efficient method."
Two ablations isolate the key ingredients. The first replaces the joint adversary with $m$ independent per-attribute adversaries, and we measure both cost and leakage of attribute combinations. The second drops the shared perturbation, computing the privacy loss on clean features only.
Why not just…?
Why not just erase attribute directions from the frozen features with a linear projection, then add Robustness Tokens for robustness? This is the most serious threat. It costs almost nothing and could match us against linear probes. Two things limit it. Linear erasure removes only linearly decodable information, and Zhang et al.'s results suggest high-overlap attributes are entangled beyond a few directions. And bolting erasure onto a separately trained robust model is exactly the naive combination that ARPRL found compounds clean-accuracy loss. We run this pipeline as a baseline and evaluate it with MLP probes trained afterwards. If it matches our privacy–utility–robustness curve against nonlinear probes, we will report that the token privacy objective is unnecessary.
Why not just use plain low-rank adapters with the privacy loss? FullLoRA-AT reports that plain low-rank adapters fail to deliver robustness unless LayerNorm is also adapted. They may still suffice for privacy, so we include both plain and LayerNorm-adapted variants. Tokens are preferred only if they reach a similar trade-off at lower cost, or if the adapters hurt utility more.
Why not just train one adversary per attribute, as in existing multi-adversary methods? Hiding each attribute alone does not guarantee that their combination is hidden. Per-attribute adversaries also scale linearly in heads, and the multi-adversary setup in EIGAN computes gradients across every network at every step. Our first ablation reports combination-level leakage, measured as probe accuracy on the joint configuration above its majority-class rate. If per-attribute adversaries leave no more joint leakage than our design, the joint head reduces to a cost-saving convenience.
Proposed Experiments and Decisive Checks
Privacy is measured as the balanced accuracy of fresh linear and two-layer MLP probes on each attribute and on the joint configuration, plus age mean absolute error on UTK-Face. Utility is measured as clean accuracy on target attributes and identity retrieval. Robustness is measured as utility under PGD-20 and AutoAttack at standard $\ell_\infty$ budgets. Cost is measured in GPU-hours and peak memory.
The first decisive check is whether, at $m=4$, the tokens bring all probes within a few points of chance, keep clean utility within the gap the robustness-only tokens show, and keep robust utility near theirs. If attribute leakage stays high under MLP probes, the tokens do not suffice. The second check is the cost curve, which should stay nearly flat from $m=1$ to $m=6$ while the multi-adversary baseline grows. The third check is mechanistic. If hiding attributes requires suppressing massive activations and robustness drops accordingly, we will have quantified the tension rather than resolved it, and we will report it as a negative result. The fourth check is whether multi-attribute feature reliability predicts per-target utility loss with correlation comparable to Zhang et al.'s 0.68. Finally, a white-box attack through the tokens bounds how much the defense depends on keeping them secret.
References
- PrivacyNet: Semi-Adversarial Networks for Multi-Attribute Face Privacy
- Can we Generalize and Distribute Private Representation Learning?
- FlowSAN: Privacy-Enhancing Semi-Adversarial Networks to Confound Arbitrary Face-Based Gender Classifiers
- Privacy-Aware Recommendation with Private-Attribute Protection using Adversarial Learning
- Learning Robust and Privacy-Preserving Representations via Information Theory
- Interpreting Disparate Privacy-Utility Tradeoff in Adversarial Learning via Attribute Correlation
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers