Location Matters: Sparse Latent-Variance Attacks for Face Privacy against Diffusion Personalization

Shuting Yang, Shuxin Li, Linyan Dai, Xinwei Zhang, Dong Huang, Renyang Liu, See-Kiong NgarXiv 2026 · arXiv

TL;DR

SEAL (Sparse Encoder Attack on Latent Variance) is a new proactive defense that protects your face from being "learned" by AI diffusion models. By intelligently perturbing only 5% of an image's pixels, it maintains near-perfect visual quality while effectively preventing tools like DreamBooth from successfully replicating your identity.

What Problem Does This Paper Address?

Text-to-image diffusion models have a "personalization" problem. Techniques like DreamBooth and LoRA allow anyone to take a handful of photos of you and fine-tune a model to generate your likeness in any context—often without your consent.

To fight back, researchers have developed "proactive" defenses: adding invisible adversarial noise to photos before they are uploaded. However, current methods face a frustrating trade-off. "Dense" attacks that modify every pixel often create visible, ugly artifacts (like color shifts or graininess), while "quality-aware" attacks are often too weak to actually stop a determined AI from learning the face. The authors of this paper argue that location matters: you don't need to change every pixel if you pick the right ones.

Methods

SEAL introduces a sparse adversarial framework that focuses on three key components:

  1. Differentiable Top-k Masking: Instead of applying noise everywhere, SEAL uses a binary mask to select exactly $k$ pixels (set to 5% of the image). Because selecting pixels is a discrete, non-differentiable operation, they use a "straight-through estimator" to jointly optimize both the values of the noise and the locations where that noise is placed.
  2. Latent Variance Disruption: The attack targets the Variational Autoencoder (VAE) used by models like Stable Diffusion. By maximizing a Laplace-based loss on the latent variance, SEAL "confuses" the encoder, making the identity-consistent signals that DreamBooth relies on much harder to extract.
  3. Saliency-Guided Dispersion: If the optimization finds that the eyes are the most effective place to put noise, it might cluster all 5% of the pixels there, making the attack visible. SEAL uses a face-parsing model (BiSeNet) to create a saliency map. It then applies a penalty to prevent noise from clustering too densely in sensitive areas like the eyes, nose, and mouth, forcing the perturbations to spread out across less noticeable regions.
Figure 2: Overview of SEAL. Given a clean image x, SEAL jointly optimizes a sparse binary mask m and a bounded perturbation δ to disrupt the VAE latent variance. A saliency-guided dispersion regularizer discourages the selected pixels from clustering in perceptually salient facial regions.

Datasets

The authors evaluated SEAL on two standard large-scale facial datasets:

  • CelebA-HQ: A high-quality dataset of 30,000 celebrity images.
  • VGGFace2: A massive dataset containing over 3.3 million images across 9,131 identities.

For the experiments, they selected 50 identities from each dataset, using 4 images per identity for the protection task.

Key Insights and Ideas

  • The 5% Sweet Spot: The authors discovered a "phase transition" in protection effectiveness. As shown in their ablation studies, protection strength jumps significantly once the sparsity ratio hits 5%, suggesting this is the critical budget needed to disrupt identity acquisition.
  • Sparsity as a Quality Feature: By leaving 95% of the pixels untouched, SEAL avoids the global color shifts and "haze" common in dense adversarial attacks.
  • Targeting the VAE is Efficient: Because SEAL only needs to backpropagate through a frozen VAE encoder (rather than the entire U-Net denoising process), it is significantly faster and more memory-efficient than many existing baselines.
Figure 1: By modifying only a small fraction of pixels, SEAL preserves the visual appearance of the protected face image while disrupting identity reproduction after diffusion person- alization. Compared with existing protection methods, SEAL achieves both high image quality and strong protection ef- fectiveness.

Key Results

SEAL outperformed five major baselines (including Anti-DreamBooth and Mist) across nearly every metric:

  • Image Quality: It achieved a PSNR (Peak Signal-to-Noise Ratio) exceeding 41 dB and an SSIM of 0.97, meaning the protected images are virtually indistinguishable from the originals to the human eye.
  • Protection Strength: After "protecting" images with SEAL, the identity similarity (ISM) of AI-generated faces dropped to as low as 0.11, and the face detection failure rate (FDFR) rose to 0.93 for certain prompts. Essentially, the AI either failed to generate a face at all or generated someone else entirely.
  • Transferability: The protection isn't just for one specific model. Noise generated using Stable Diffusion 2.1 as a surrogate successfully "fooled" personalization on SD 1.5 and SDXL as well.
Visual comparison showing that while baselines like Anti-DB and Mist create visible grain, SEAL's perturbations are nearly invisible, yet the resulting AI generations (Gen. P2/P3) are completely broken.

Limitations and Future Work

While SEAL is highly effective, the authors note two main hurdles:

  1. Preprocessing Vulnerability: Like most pixel-level adversarial attacks, SEAL is sensitive to common image transformations. Operations like heavy JPEG compression, Gaussian blurring, or aggressive resizing can "wash away" the sparse noise, potentially restoring the AI's ability to learn the identity.
  2. Encoder-based Personalization: SEAL targets fine-tuning methods (like DreamBooth). It does not currently protect against "encoder-based" personalization methods like IP-Adapter FaceID, which bypass the VAE fine-tuning stage.

Improving the robustness of sparse perturbations against real-world image processing remains a key direction for future research.

Summary updated

Abstract

Text-to-image diffusion models can learn and reproduce an individual's identity from only a few reference photos, creating risks of unauthorized and misleading content generation. Proactive defenses address this risk by adding adversarial perturbations to facial images before release, thereby disrupting identity acquisition during downstream personalization. However, existing methods typically rely on dense $L_\infty$-bounded perturbations, whose accumulated distortions can become visible around facial structures. Recent methods improve protected-image quality, but often at the cost of reduced protection effectiveness. In this paper, we propose Sparse Encoder Attack on Latent Variance (SEAL), a sparse adversarial protection framework that modifies only 5% of the spatial locations in a facial image. SEAL combines a differentiable top-k masking scheme that jointly optimizes perturbation locations and values with a variance-oriented objective over the frozen VAE encoder's latent distribution to disrupt identity-consistent representations. A saliency-guided dispersion penalty further discourages perturbation concentration in perceptually salient facial regions. Experiments on CelebA-HQ and VGGFace2 show that SEAL achieves the best protected-image quality among the compared methods, exceeding 41 dB PSNR with 0.97 SSIM and at most 0.07 LPIPS, while reducing identity similarity to at most 0.15 after diffusion personalization. These results demonstrate that sparse location optimization enables strong identity protection while preserving high visual quality. Code and data are available at https://github.com/Lvxan/SEAL
BibTeX
@misc{yang2026location,
  author = {Shuting Yang and Shuxin Li and Linyan Dai and Xinwei Zhang and Dong Huang and Renyang Liu and See-Kiong Ng},
  title = {{Location Matters: Sparse Latent-Variance Attacks for Face Privacy against Diffusion Personalization}},
  year = {2026},
  eprint = {2611.14254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14254},
}
Figure