Latent Transfer Attack: Adversarial Examples via Generative Latent Spaces

Eitan Shaar, Ariel Shaulov, Yalcin Tur, Gal Chechik, Ravid Shwartz-Ziv

Abstract

Adversarial attacks are a central tool for probing the robustness of modern vision models, yet most methods optimize perturbations directly in pixel space under $\ell_\infty$ or $\ell_2$ constraints. While effective in white-box settings, pixel-space optimization often produces high-frequency, texture-like noise that is brittle to common preprocessing (e.g., resizing and cropping) and transfers poorly across architectures. We propose LTA (Latent Transfer Attack), a transfer-based attack that instead optimizes perturbations in the latent space of a pretrained Stable Diffusion VAE. Given a clean image, we encode it into a latent code and optimize the latent representation to maximize a surrogate classifier loss, while softly enforcing a pixel-space $\ell_\infty$ budget after decoding. To improve robustness to resolution mismatch and standard input pipelines, we incorporate Expectation Over Transformations (EOT) via randomized resizing, interpolation, and cropping, and apply periodic latent Gaussian smoothing to suppress emerging artifacts and stabilize optimization. Across a suite of CNN and vision-transformer targets, LTA achieves strong transfer attack success while producing spatially coherent, predominantly low-frequency perturbations that differ qualitatively from pixel-space baselines and occupy a distinct point in the transfer-quality trade-off. Our results highlight pretrained generative latent spaces as an effective and structured domain for adversarial optimization, bridging robustness evaluation with modern generative priors.

Introduction

Adversarial attacks—small, intentional tweaks to an image that cause a model to misclassify it—are a primary way we test the "robustness" of vision models. However, standard attacks have a "transferability" problem. If you generate an attack on a ResNet (a CNN), it often fails to fool a Vision Transformer (ViT). This happens because most attacks optimize individual pixels directly, creating high-frequency, texture-like noise that exploits the specific "quirks" of one model's architecture. Furthermore, this noise is brittle; simply resizing or cropping the image often "breaks" the attack. For an attack to be truly transferable, it needs to target the broader semantic and structural features that all good vision models share, rather than just pixel-level artifacts.

Methods

The core innovation of LTA is shifting the "battleground" from pixel space to the generative latent space of a Variational Autoencoder (VAE), specifically the one used in Stable Diffusion.

  1. Latent Optimization: Instead of tweaking pixels, LTA tweaks a "latent code." This code is then passed through a frozen VAE decoder to produce the final adversarial image. Because the decoder is trained to produce natural-looking images, it acts as a filter that naturally biases perturbations toward low-frequency, spatially coherent structures.
  2. Soft $\ell_\infty$ Constraint: To ensure the image doesn't change too much visually, a "soft" penalty discourages the decoded image from deviating too far from the original pixels.
  3. Expectation Over Transformations (EOT): To make the attack survive real-world preprocessing, the latent code is optimized while simulating random resizing, cropping, and interpolation.
  4. Periodic Latent Smoothing: To prevent the optimization from creating "artifacts" in the latent space itself, a small Gaussian blur is periodically applied to the latent perturbation, keeping the signal "clean."

Experimental Results

The researchers evaluated LTA using a standard ImageNet-compatible dataset consisting of 1,000 images. They tested the attack's success across a diverse suite of target architectures, including various CNNs (ResNet, VGG, MobileNet, Inception) and Vision Transformers (ViT, PiT, Swin).

Superior Transferability

As shown in Table 1, when using a ResNet-50 as the surrogate model, LTA significantly improves the attack success rate (ASR) on Transformer targets compared to existing baselines. On average, LTA achieves an 89.9% ASR across ViT targets when using a ResNet-50 surrogate, outperforming the next best baseline (P2FA at 83.4%) by a significant margin.

Table 1: Transfer attack success rates (ASR) across various CNN and ViT target architectures using different surrogate models.

Spectral Analysis and Mechanism

Frequency domain analysis confirms that LTA concentrates its energy in low-frequency bands, whereas standard pixel-space attacks are spread across the entire spectrum. This low-frequency bias is a key factor in its robustness and transferability. By attacking the latent space, LTA targets features that are more fundamental to the image itself, which both CNNs and ViTs rely on for classification.

Figure 1: Average power spectrum of perturbations across normalized frequencies, showing LTA's concentration in low-frequency bands.

Robustness to Defense

LTA was remarkably effective against "purification" defenses. Against the DiffPure defense (which uses diffusion to clean images), LTA outperformed previous state-of-the-art attacks by up to 34.3 points. Because LTA's perturbations look like natural parts of the image structure, these defenses struggle to distinguish the "attack" from the "content."

Conclusion

LTA demonstrates that pretrained generative latent spaces provide an effective and structured domain for adversarial optimization. By leveraging the inductive bias of a VAE decoder, LTA produces spatially coherent, low-frequency perturbations that bridge the transferability gap between different architectures like CNNs and ViTs. This approach not only improves attack success rates but also enhances robustness against common image preprocessing and advanced purification defenses.

Summary updated

Abstract

Adversarial attacks are a central tool for probing the robustness of modern vision models, yet most methods optimize perturbations directly in pixel space under $\ell_\infty$ or $\ell_2$ constraints. While effective in white-box settings, pixel-space optimization often produces high-frequency, texture-like noise that is brittle to common preprocessing (e.g., resizing and cropping) and transfers poorly across architectures. We propose $\textbf{LTA}$ ($\textbf{L}$atent $\textbf{T}$ransfer $\textbf{A}$ttack), a transfer-based attack that instead optimizes perturbations in the latent space of a pretrained Stable Diffusion VAE. Given a clean image, we encode it into a latent code and optimize the latent representation to maximize a surrogate classifier loss, while softly enforcing a pixel-space $\ell_\infty$ budget after decoding. To improve robustness to resolution mismatch and standard input pipelines, we incorporate Expectation Over Transformations (EOT) via randomized resizing, interpolation, and cropping, and apply periodic latent Gaussian smoothing to suppress emerging artifacts and stabilize optimization. Across a suite of CNN and vision-transformer targets, LTA achieves strong transfer attack success while producing spatially coherent, predominantly low-frequency perturbations that differ qualitatively from pixel-space baselines and occupy a distinct point in the transfer-quality trade-off. Our results highlight pretrained generative latent spaces as an effective and structured domain for adversarial optimization, bridging robustness evaluation with modern generative priors.
BibTeX
@misc{shaar2026latent,
  author = {Eitan Shaar and Ariel Shaulov and Yalcin Tur and Gal Chechik and Ravid Shwartz-Ziv},
  title = {{Latent Transfer Attack: Adversarial Examples via Generative Latent Spaces}},
  year = {2026},
  eprint = {2603.06311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.06311},
}
Figure