Cheap Style Adversaries for Semantic Robustness of Frozen Backbones

Image classifiers can be fooled without adding pixel noise. An attacker can recolor a car, restyle a texture, or edit a photo with a text instruction. The result looks perfectly natural to a person, yet the model's prediction flips. The usual defense, adversarial training, retrains the model on worst-case inputs found by an inner search. For these "unrestricted semantic" attacks that defense is close to unusable. Each search step would run a diffusion model or a generative network, retraining a large vision backbone that way is out of reach, and full retraining tends to lower accuracy on ordinary images anyway.

This note proposes a cheaper route, which we call feature-statistic adversarial tokens. We keep a pretrained vision backbone frozen and train only a handful of extra input vectors. Their training adversary is deliberately cheap: it shifts the per-channel mean and spread of the backbone's early-layer activations, which is roughly what a style or color change does, plus a global color transform in a perceptual color space. Nothing is generated by a diffusion model or GAN during training. The bet is that most semantic attacks succeed by pushing the representation along a small set of appearance directions. If that holds, hardening those directions alone should transfer to expensive attacks such as language-guided diffusion edits, at roughly the cost of a few hundred optimization steps and without hurting clean accuracy. If it does not hold, the experiments will show where the expensive attacks leave that appearance subspace.

Core Claim

The claim has two parts, and both can be tested.

The first part is a diagnosis. When a semantic attack fools a frozen foundation backbone, most of the displacement it causes in the backbone's intermediate features lies in a low-dimensional subspace. That subspace can be estimated from benign appearance transformations alone. The idea is plausible because Substance or Style shows that foundation embeddings encode which synthetic transformation was applied: probes identify the transformation with over 93% accuracy for most models, and CLIP reaches 86.24% on unseen styles. Appearance information is therefore present and linearly readable, separate from content.

The second part is a remedy. A learned-token defense of the kind introduced by Robustness Tokens can be trained against a differentiable adversary confined to that subspace, and it will then hold up against attacks it never saw: Instruct2Attack, ColorFool, and colorization or texture attacks. That defense keeps the backbone frozen and trains about ten prepended vectors in roughly 200 steps against pixel noise. The adversary we plug in borrows the feature-statistic parameterization of Towards Feature Space Adversarial Attack by Style Perturbation. That paper used this parameterization to attack models, driving an adversarially trained model to 1.25%. Here it becomes a cheap training adversary.

The proposal also gives an explicit answer to a problem that TART leaves open. TART improves clean accuracy by training only on perturbations that run along the data manifold (CIFAR-10 clean accuracy rises from 81.76% to 83.73%). Semantic attacks are almost entirely along-manifold, though, so TART's rule would accept all of them, including ones that change what the image depicts. We split the along-manifold directions instead. Appearance statistics are directions the classifier should ignore by construction. Spatial content is left alone.

Research Program

The first stage is diagnostic. We take OpenCLIP and DINOv2 backbones and estimate an appearance subspace at several early and middle transformer blocks. For each image, we apply benign appearance transformations (color affine maps in Lab space, AdaIN-style restyling, brightness and saturation changes) and stack the resulting changes in per-channel token mean and standard deviation. We then compute the leading principal directions of that stack.

Next we generate real semantic attacks against the undefended backbone. These are Instruct2Attack (Instruct2Attack), ColorFool (ColorFool), and the cAdv and tAdv attacks (Unrestricted Adversarial Examples via Semantic Manipulation). For each attack we measure what fraction of the induced feature change falls inside the appearance subspace. If that fraction is low for the diffusion attacks, the cheap-adversary idea fails for them, and we learn that before building anything further.

The second stage builds the defense. The trainable parameters are a small set of prepended tokens $r$. The adversary has two kinds of parameters. One is a per-channel scale and shift $(\gamma, \beta)$ applied to the patch-token activations at chosen blocks. The other is a global $3\times 3$ affine map $A$ on the image's Lab color channels.

Training follows the dual objective of Robustness Tokens. Let $f$ be the frozen backbone and $f_r$ the backbone with the tokens attached. Let $T_{\gamma,\beta,A}$ denote applying the adversary's perturbation. We minimize

$$ \mathcal{L}(r) = \mathbb{E}_x\Big[\, d\big(f_r(x), f(x)\big) + \lambda \max_{(\gamma,\beta,A)\in \mathcal{S}} d\big(f_r(T_{\gamma,\beta,A}(x)), f(x)\big) \Big], $$

where $d$ is cosine distance between output features and $\lambda$ weights the correction term. The first term keeps the tokens from changing clean features, which protects clean accuracy. The second term pulls attacked features back to the clean ones.

The set $\mathcal{S}$ bounds $(\gamma,\beta,A)$ by the range observed under benign transformations, projected onto the estimated appearance subspace. This bound is what keeps the threat label-preserving. A shift in channel statistics changes appearance, not object layout. The inner maximization takes a few gradient steps over at most a few hundred scalars. Its cost is therefore about the same as the pixel-noise inner loop used by Robustness Tokens, and orders of magnitude below a diffusion-in-the-loop attack.

The third stage is evaluation under attacks the defense never trained on, including fully adaptive white-box versions. Robustness Tokens relied partly on keeping the token values secret. We drop that assumption: the attacker gets the tokens and backpropagates through them.

Baselines

The naive reference point is the frozen backbone with a linear probe, which we also run zero-shot for CLIP. SSCD reports zero-shot CLIP at 6.0% robust accuracy against Instruct2Attack on a 5,000-image ImageNet subset. We regenerate these attacks on the same subset protocol to confirm that our setup reproduces this weakness.

The strongest existing method on this exact question is SSCD. It builds a GPT-4 and WordNet concept dictionary, sparse-codes the frozen CLIP embedding onto that dictionary with LASSO, and classifies the sparse codes with a linear layer. It is the only method in the file that was designed against semantic attacks, uses a frozen backbone, and reports an ImageNet number: 29.9% robust accuracy against Instruct2Attack on the 5,000-image subset. We run it on the same OpenCLIP backbone, the same images, and the same attack generation settings.

Beating SSCD means two things. Robust accuracy against Instruct2Attack must be clearly above 29.9%. Clean accuracy must stay within about one point of the frozen linear probe. We expect a cost advantage as well, since our defense needs one forward pass per image while SSCD solves a LASSO problem per sample. We report wall-clock time per image so that comparison is concrete.

For ImageWoof, no reported numerical result was found on Lacuna for SSCD, so we produce those numbers ourselves.

The second strong competitor is off-the-shelf Robustness Tokens trained against pixel-norm noise. On DINOv2-Giant it keeps clean accuracy at 85.2% versus 85.6% and recovers white-box pixel-attack accuracy from 0.3% to 63.1%. It shares our parameter count and training budget exactly, so it isolates the question that matters most: does changing the adversary from pixel noise to feature statistics change semantic robustness? We give it the same 200-step budget, the same token count, and the same tuning budget.

TART cannot be run directly on frozen ViTs. Its pixel-space tangent matrices are precomputed per static image. We therefore include a feature-space analog: tokens trained against pixel-noise adversaries filtered by TART's top-50% tangential rule. This tests whether "along the manifold" alone is the right selection criterion.

Three ablations isolate the key ingredient. The first samples $(\gamma,\beta,A)$ at random from $\mathcal{S}$ instead of maximizing over it. The second drops the projection onto the estimated appearance subspace. The third removes the color affine $A$, leaving only the feature-statistic adversary.

Why not just…?

Why not just generate Instruct2Attack and ColorFool examples offline and train the tokens on them? This is the most serious threat, and we run it head to head with the same token count, step count, and number of training images. The file gives two reasons to expect it to fall short. First, its examples are computed against the undefended model, so it is a fixed rather than adaptive adversary. Second, diffusion-based generation is the cost bottleneck that the dossier flags for Instruct2Attack and SemanticAdv. If offline training matches our robustness, the contribution reduces to cost savings, and we would say so. We report both robustness and the total GPU-hours each approach needs, including attack generation.

Why not just apply random color jitter and style transfer with the invariance loss? Color Backdoor reports that aggressive color augmentation destabilizes training and lowers clean accuracy. However, that finding concerns full-model training, not a frozen backbone with an invariance term. SemanticAdv and the style-perturbation work indicate that random interpolation rarely finds the attack directions; gradient-guided search does. The random-sampling ablation decides this question directly. If random sampling performs as well as maximization, the inner loop is unnecessary and the method gets even cheaper.

Why not just project the appearance subspace out of the final embedding, with no training at all? If the diagnostic in the first stage succeeds, this zero-cost fix becomes a real competitor, and we include it as a baseline. Our reason to doubt it is that an adaptive attacker who knows the projection can work at the input. Through the nonlinear backbone, that attacker can steer the final embedding along content directions the projection keeps. The tokens act inside the network, at the blocks where appearance statistics first enter. The adaptive white-box evaluation of the projection baseline is the experiment that tells these apart.

Why not just use fast pixel-norm adversarial training? The style-perturbation paper reports that an adversarially trained ImageNet model fell to 1.25% under style attacks. The pixel-noise Robustness Tokens baseline measures how much pixel-norm hardening transfers in our exact setting.

Proposed Experiments

The datasets are the 5,000-image ImageNet subset with Instruct2Attack, following SSCD; ImageWoof for fine-grained dog breeds; and CIFAR-10 and Places365 for ColorFool. Backbones are OpenCLIP and DINOv2 (Base and Giant). We report clean accuracy, robust accuracy for each attack family, and AutoAttack accuracy under the RobustBench protocol (RobustBench) to check that pixel-noise robustness is not lost. We also report training GPU-hours and inference latency.

Four checks are decisive. The first is the subspace-capture fraction from the diagnostic stage, broken down by attack family. The second is held-out transfer: the defense never sees Instruct2Attack, cAdv, or tAdv during training. The third is the full white-box adaptive attack with known token values. The fourth is the clean-accuracy change relative to the frozen probe.

Two outcomes count as decisive. A positive result is robust accuracy above SSCD's 29.9% under adaptive Instruct2Attack, with clean accuracy within one point of the probe. A negative result is a low capture fraction for the diffusion attacks together with failed transfer. That would show language-guided edits change content-level directions that no appearance-only adversary can stand in for.