On the Robustness of Latent Diffusion Models
Lists
Only you see this
Abstract
Latent diffusion models have achieved state-of-the-art performance on a variety of generative tasks, such as image synthesis and image editing. However, the robustness of latent diffusion models is not well studied. Previous works only focus on the adversarial attacks against the encoder or the output image under white-box settings, regardless of the denoising process. Therefore, in this paper, we aim to analyze the robustness of latent diffusion models more thoroughly. We first study the influence of the components inside latent diffusion models on their white-box robustness. We find out that the denoising process, especially the Resnet, is the most vulnerable to adversarial attacks. In addition to white-box scenarios, we evaluate the black-box robustness of latent diffusion models via transfer attacks, where we consider both prompt-transfer and model-transfer settings and possible defense mechanisms. We conclude that the adversarial vulnerability is inherited with the development of Stable Diffusion models, and the adversarial attacks are still effective when possible defenses are present. Additionally, analyzing the robustness of latent diffusion models needs a comprehensive benchmark dataset, which is missing in the literature. Therefore, to facilitate the research on the robustness of latent diffusion models, we propose two automatic dataset construction pipelines for two kinds of image editing models and release the whole dataset.
BibTeX
@misc{zhang2023robustness,
author = {Jianping Zhang and Zhuoer Xu and shiwen cui and Changhua Meng and Weibin Wu and Michael Lyu},
title = {{On the Robustness of Latent Diffusion Models}},
year = {2023},
eprint = {2306.08257},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2306.08257},
}