High-Frequency Anti-DreamBooth: Robust Defense against Personalized Image Synthesis

Takuto Onikubo, Yusuke Matsui

Abstract

Text-to-image generative models have been increasingly misused to create unauthorized malicious images of individuals. While existing defenses like Anti-DreamBooth add adversarial noise to protect images, these perturbations are often fragile and can be removed via adversarial purification methods like DiffPure. We propose High-Frequency Anti-DreamBooth (HF-ADB), a novel defense that concentrates strong adversarial perturbations in high-frequency image regions. By targeting edges and textures, HF-ADB achieves robustness against purification while maintaining visual quality. Experimental results demonstrate that HF-ADB effectively hinders malicious image generation even after adversarial cleaning, offering a more resilient solution for personal image protection.

Introduction

The rise of text-to-image models such as Stable Diffusion has introduced significant social challenges, particularly through "personalization" techniques like DreamBooth. These methods allow attackers to fine-tune models on a small set of unauthorized images to generate realistic deepfakes. To counter this, researchers have proposed adversarial defenses that "poison" images with subtle noise, causing fine-tuning to fail. However, these defenses are often vulnerable to adversarial purification—preprocessing steps like bilateral filtering or diffusion-based cleaning (e.g., DiffPure) that scrub low-magnitude noise.

Methodology: High-Frequency Targeting

High-Frequency Anti-DreamBooth (HF-ADB) addresses the fragility of previous methods by strategically distributing adversarial noise. Instead of a uniform noise budget, HF-ADB focuses intense perturbations on high-frequency areas, such as edges and complex textures, which are naturally more resilient to purification and less sensitive to human perception.

The HF-ADB pipeline involves three key steps:

  1. Edge Detection: A $3 \times 3$ Laplacian filter is applied to the input image to identify high-frequency components.
  2. Mask Generation: A binary mask is created by thresholding the edge values, typically isolating 3-5% of the image.
  3. Optimization via ASPL: The system employs Alternating Surrogate and Perturbation Learning (ASPL) to optimize noise against a surrogate model. A spatially varying noise budget is used: a baseline perturbation ($\eta$) for the whole image and a significantly stronger perturbation ($\eta_{mask}$) for the masked edge regions.
The HF-ADB procedure: Laplacian filtering identifies edges to create a mask for targeted, high-intensity adversarial perturbations.

Experimental Setup

The effectiveness of HF-ADB was evaluated using the VGGFace2 dataset. The researchers sampled 400 images from 50 individuals, using four images per subject for protection and four as reference images for surrogate model training. All images were processed at a $512 \times 512$ resolution.

Results and Discussion

Qualitative analysis reveals that HF-ADB is significantly more robust than standard Anti-DreamBooth. When subjected to purification via bilateral filters, the baseline defense is often neutralized, allowing the generation of clean, realistic images. In contrast, images protected by HF-ADB continue to produce "unnatural patterns" and severe artifacts in the generated output, even after purification attempts.

HF-ADB maintains its defensive capability even after adversarial purification. The generated images (bottom row) remain distorted, indicating the model failed to learn the subject's identity.

A key insight from the study is the limitation of standard computer vision metrics. While visual results clearly showed the success of the defense, automated metrics like Face Detection Success Ratio often failed to capture the qualitative degradation of the generated images. This highlights a need for more specialized evaluation frameworks for adversarial defenses in generative AI.

Limitations and Future Work

While effective for human faces, HF-ADB showed reduced efficacy when applied to artworks (e.g., paintings by Van Gogh). The inherent texture and color complexity of artistic styles may naturally absorb or mask the adversarial noise, making it less disruptive to the fine-tuning process. Additionally, while the noise is concentrated on edges, it may still be perceptible under close scrutiny. Future research will focus on improving the imperceptibility of the noise and extending the defense's robustness across diverse image domains.

Summary updated

Abstract

Recently, text-to-image generative models have been misused to create unauthorized malicious images of individuals, posing a growing social problem. Previous solutions, such as Anti-DreamBooth, add adversarial noise to images to protect them from being used as training data for malicious generation. However, we found that the adversarial noise can be removed by adversarial purification methods such as DiffPure. Therefore, we propose a new adversarial attack method that adds strong perturbation on the high-frequency areas of images to make it more robust to adversarial purification. Our experiment showed that the adversarial images retained noise even after adversarial purification, hindering malicious image generation.
BibTeX
@inproceedings{onikubo2024high,
  author = {Takuto Onikubo and Yusuke Matsui},
  title = {{High-Frequency Anti-DreamBooth: Robust Defense against Personalized Image Synthesis}},
  booktitle = {DarkSide of GenAIs and Beyond @ ECCV},
  year = {2024},
  url = {https://openreview.net/forum?id=uRqU6l1GmH},
}
Figure