Honest Privacy Filters on Frozen Encoders Without the Alternating Adversary Loop

Many systems now send the internal features of a large pretrained model, rather than raw data, to an untrusted server. A smart-glasses assistant, for example, ships its visual tokens to a cloud language model. Those features leak private traits such as gender, location or identity. The standard fix is a minimax privacy filter: a small module is trained to keep the features useful for the task while a second network, the "adversary," simultaneously tries to predict the private trait from them. The filter learns to defeat it.

This game has the same cost problem as adversarial training. Every filter update needs an adversary that is close to its best response. Practitioners therefore run only a few adversary steps per filter step, and nobody checks whether the reported privacy survives an attacker trained to convergence afterward. Adversarial robustness has a known failure of this kind, called gradient masking: a weak inner attack makes a defense look strong. We think privacy filters suffer the same illusion.

We propose to replace the iterative adversary with one that is solved exactly on every minibatch. That adversary is a ridge-regression probe over random nonlinear features, and its solution is cheap to compute and to differentiate through. We pair it with a gradient-projection step that stops privacy updates from undoing task-relevant directions. The claim is that this yields filters whose privacy holds up against fresh, strong attackers, at a training cost close to one adversary step. It also lets us separate the unavoidable accuracy loss (set by how correlated the private trait is with the task label) from the avoidable loss caused by conflicting gradients.

Core Claim

The three input directions line up as follows. Minimax privacy is the object of study. The cost of the inner maximization, the central problem in adversarial-training overhead, is the lens. Clean-accuracy degradation is the quantity we try to drive down to its theoretical floor.

Our claim has two parts. First, filters trained with cheap alternating adversaries overstate their privacy when audited by an attacker trained to convergence after the filter is frozen. Second, an exactly solved random-feature adversary closes most of that gap at a small fraction of the cost of running many adversary steps.

Concretely, let $h = g_\phi(z)$ be the filter output for frozen-backbone features $z$, and $\psi(h) \in \mathbb{R}^m$ a fixed random Fourier feature map. On each minibatch we split the examples into halves A and B. We fit the attacker in closed form on half A:

$$W^\star(\phi) = (\Psi_A^\top \Psi_A + \gamma I)^{-1} \Psi_A^\top S_A,$$

where $\Psi_A$ stacks the random features and $S_A$ the one-hot private attributes, and $\gamma$ is a ridge penalty. The filter then minimizes

$$\mathcal{L}_{\text{util}}(\phi) - \lambda\, \ell\big(\Psi_B W^\star(\phi), S_B\big),$$

where $\ell$ is the held-out attacker loss on half B and $\lambda$ trades utility against privacy. Gradients flow through the linear solve, which costs $O(m^3)$ per step and is negligible for $m$ of a few thousand.

Before applying the privacy gradient, we check it against the utility gradient. When the two conflict (negative inner product), we project the privacy gradient onto the orthogonal complement of the utility gradient. This is the Gradient Orthogonal Projection step from Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks, which that paper applied to post-hoc unlearning rather than to a live privacy game.

Relation to Prior Work

Several papers motivate the gap by relying on trained adversaries whose strength is never audited. The original minimax filter work, Preserving Privacy of Continuous High-dimensional Data with Minimax Filters, set up the contributor–aggregator–adversary game and drove private traits to chance on Genki and Enterface. GEN in Protecting Sensory Data against Sensitive Inferences scrubs gender from motion sensors with an on-device autoencoder. The Deep Variational Privacy Funnel uses six-step alternating block coordinate descent over several networks. All three pay for an inner adversary. The research file also flags adversary mismatch as their main weakness: guarantees hold only for the attacker class used in training. We attack exactly that weakness, and the cost of the inner loop, together.

Two recent works are the closest systems. Defending Wearable VLMs Against Private Attribute Inference (TGAP) trains a residual adapter on visual tokens against an auxiliary attacker decoder plus a language-model uncertainty loss. Privacy Beyond Pixels (SPLAVU) places an anonymizing adapter after a frozen VideoMAE. Both already achieve parameter efficiency, so we do not claim novelty there. What we change is how the adversary inside the training loop is computed, and how its gradient is combined with the utility gradient. The adapter itself we keep.

One paper constrains what counts as success. MaSS proves that achievable suppression is bounded by the mutual information between the private attribute and the task label. Gradient projection therefore cannot eliminate accuracy loss when the two are coupled. We treat that bound as a floor to measure against, not a limit to beat.

The adversarial-robustness side supplies the analogy and one diagnostic. Robustness Tokens shows that frozen transformers can be steered cheaply, but in only 200 steps and only for evasion robustness. Mitigating Adversarial Effects Through Randomization is the canonical case of a defense that looks strong until a stronger, transformation-averaging attacker is used. That is the failure pattern we expect to find in privacy filters.

Research Program

The work has three parts.

The audit. We take existing filter-training recipes and vary the number of adversary steps per filter step, $k \in \{1, 5, 20\}$. After training, we freeze each filter. We then train a battery of fresh attackers to convergence on held-out filtered features: linear probes, deep multilayer perceptrons, gradient-boosted trees, and, for the vision-language setting, a transformer decoder over tokens. We report the gap between leakage measured by the training-time adversary and the worst leakage across the battery. This "privacy illusion gap" is the primary quantity.

The method. We build the closed-form random-feature adversary with conflict-aware projection on top of TGAP-style and SPLAVU-style adapters, so that only the adversary and the gradient combination differ from those systems. We study how the random-feature width $m$ and bandwidth affect the illusion gap. The expectation is that a wider, richer feature space approximates stronger attacker classes, with diminishing returns once the best fresh attacker is matched.

The accounting. For each filter we compute three quantities: its utility loss; an estimate of the MaSS floor, from the empirical mutual information between the private attribute and the label; and the residual, which we attribute to avoidable gradient conflict. The projection ablation tests whether that residual shrinks.

As a secondary diagnostic we ask whether privatization changes evasion robustness. We run PGD (projected gradient descent, a standard iterative input attack) on the utility task, with and without the filter. This is not the headline, but no paper in the file measures it.

Baselines

Do-nothing reference. The unprotected backbone. On the wearable-VLM benchmark, Defending Wearable VLMs reports 56.7% privacy inference accuracy and 76.6% relaxed utility accuracy. On MotionSense, Protecting Sensory Data reports gender accuracy above 90% from raw data and activity accuracy of 95%.

Strongest existing method: TGAP on the VLM benchmark. It reports privacy inference falling to 7.4% with utility at 74.4%. It is the strongest competitor because it already combines a lightweight adapter, an attacker decoder and identity regularization on exactly our deployment setting. We will retrain it with its own recipe and the same adapter capacity, then audit it with our fresh-attacker battery. The file does not say how its 7.4% was measured relative to a converged post-hoc attacker. Our method beats TGAP if it achieves lower worst-case audited leakage at equal or better utility, with no more training wall-clock.

Video setting: SPLAVU on VISPR. It reports a 35% leakage reduction with utility within 1–2% of baseline, and serves as the video baseline under the same protocol.

Sensor setting: MaSS on MotionSense. It reports 0.0 normalized accuracy gain on gender and identity with 0.89 on activity. It is the reference on MotionSense, where full retraining of every attacker is cheap enough to audit exhaustively.

Fairness controls. All methods receive the same frozen backbone, data splits, adapter parameter count, hyperparameter search budget (the same number of trials over $\lambda$ and learning rate), and a GPU-hour cap per run.

Alternating-adversary ablation. The same adapter trained with a $k$-step alternating adversary, $k \in \{1, 5, 20\}$. This isolates the closed-form inner solve.

Projection ablation. Our method with the gradient projection turned off. This isolates the conflict-handling ingredient.

Linear-probe variant. Our method with $\psi$ set to the identity. This tests whether nonlinear random features are needed at all.

Why not just…?

Why not just run more adversary steps? This is the real threat. If a 20-step alternating adversary matches our audited leakage at similar wall-clock, the closed-form solve buys nothing. The audit's leakage-versus-GPU-hours curve settles it directly. We claim success only if our method sits on or beyond that curve's frontier at roughly the cost of $k \le 2$. If it does not, the paper becomes an audit paper showing that cheap alternating filters overstate privacy. That result would still be useful, but it is a smaller claim.

Why not just remove attribute directions with a linear projection of the features? A closed-form linear removal is cheaper still. Our linear-probe variant is essentially this, trained end to end. We expect it to pass linear audits and fail against the deep-attacker battery, because private and task information are entangled nonlinearly within tokens. Defending Wearable VLMs shows this entanglement defeats hard token deletion. If the linear variant survives the nonlinear audit, we will say so and recommend it.

Why not just add differential-privacy noise? Worst-case noise ignores the data distribution. Both the minimax filter paper and the variational privacy funnel argue it destroys utility on structured continuous data. We include a Gaussian-noise-on-features curve at matched audited leakage to verify this on our benchmarks, rather than assume it.

Why not just tune $\lambda$ instead of projecting gradients? Scalar reweighting only moves along one trade-off curve. Forget by Uncertainty reports that it leaves a larger retained-accuracy gap than projection: 7.06% for Q-MUL versus 3.67% for OEU on CIFAR-100. The projection ablation, run with a full $\lambda$ sweep for both variants, tests whether the same holds in a live minimax game. Because of the MaSS bound, we expect gains only on attributes weakly correlated with the task label.

Proposed Experiments

Datasets. MotionSense (activity versus gender and identity), the paired Reddit+VQA wearable-VLM benchmark with MiniCPM-V (four attribute categories), and VISPR with VideoMAE features.

Metrics. Worst-case audited attribute accuracy across the fresh-attacker battery, the privacy illusion gap, clean utility, utility loss relative to the MaSS floor estimate, training GPU-hours, and, as a diagnostic, PGD robust utility accuracy.

First decisive check. Do $k=1$ alternating filters, including our TGAP reproduction, show a substantial illusion gap on at least two of the three datasets? If they do not, the motivating premise fails, and we would report that cheap adversaries are adequate.

Second decisive check. At matched GPU-hours, does the closed-form adversary achieve lower worst-case audited leakage than every alternating setting, at no worse utility than TGAP's reported 74.4%?

Third decisive check. Does projection reduce utility loss on attribute–task pairs with low estimated mutual information, and leave it unchanged on strongly coupled pairs? That split is what the MaSS bound predicts. A uniform improvement would suggest our floor estimate is wrong.