A Training-Free Plug-and-Play Watermark Framework for Stable Diffusion

Guokai Zhang, Lanjun Wang, Yuting Su, An-An LiuarXiv 2024 · arXiv · OpenReview

TL;DR

This paper introduces MarkPlugger, a novel framework that embeds watermarks into images generated by Latent Diffusion Models (LDMs) without requiring any retraining of the LDM itself. This "plug-and-play" approach significantly reduces computational costs and enables flexible watermarking across various LDM versions, while maintaining high image quality and watermark robustness.

What Problem Does This Paper Address?

The rapid rise of powerful text-to-image generative models like Latent Diffusion Models (LDMs) has brought immense creative potential but also significant security concerns. Malicious actors can exploit these models to generate and spread harmful content, making it difficult to trace the origin of these creations and assign responsibility. Current solutions often involve retraining parts of the LDM or the entire model to embed watermarks. However, the fast-paced development and frequent updates of LDMs make this retraining process costly and time-consuming, hindering widespread adoption of watermarking. Furthermore, existing methods can be inflexible, requiring custom watermarks for each LDM variant or pre-embedding watermarks into training data, which is also resource-intensive. This paper addresses the need for a generalizable, cost-effective, and flexible watermarking solution for LDMs that doesn't require model retraining.

Methods

MarkPlugger operates as a "plug-and-play" module, meaning it can be integrated with existing LDMs without altering their core architecture or requiring fine-tuning. The key innovation lies in how it embeds watermarks within the latent space of the LDM, specifically during the denoising process.

  1. Latent Space Watermarking: Instead of modifying the image pixels or the diffusion process directly, MarkPlugger targets the intermediate latent representations. It identifies a watermark representation that is approximately orthogonal to the semantic information in the latent space. This is crucial for minimizing visual distortion.
  2. Additive Fusion: A novel additive fusion strategy is employed to combine the watermark signal with the latent code. This strategy focuses on perturbing only a single channel of the latent representation to further reduce the impact on image quality.
  3. Training-Free Watermark Encoder-Decoder: The framework utilizes a separate watermark encoder-decoder architecture. Crucially, this watermark model is trained independently of the LDM. Once trained, it can be applied to various LDMs without any retraining.
  4. Objective Function: A custom loss function is designed to balance image quality (minimizing distortion between original and watermarked images) and watermark recoverability (ensuring the watermark can be accurately extracted). This loss function considers both pixel-level and perceptual similarity metrics.

Datasets

  • COCO Subset: Used for training and evaluation, comprising 82,783 image-text pairs for training and 10,000 for inference.
  • Flickr-8k: Used to evaluate generalization to different datasets.

Key Insights and Ideas

  • Latent Space is Key: The paper demonstrates that embedding watermarks in the latent space, rather than pixel space, offers a more efficient and less disruptive approach for LDMs.
  • Orthogonality for Quality: The idea of creating a watermark representation that is "approaching orthogonal" to the semantic representation in latent space is a clever way to minimize visual artifacts.
  • Plug-and-Play Efficiency: The core insight is that a well-trained, independent watermark module can be seamlessly "plugged into" various LDM architectures without costly retraining, making watermarking much more accessible.
  • Single-Channel Perturbation: Limiting watermark embedding to a single channel of the latent code is a simple yet effective technique to preserve overall image quality.

Key Results

  • Generalization: MarkPlugger demonstrates remarkable generalization capabilities, working effectively across multiple official versions of Stable Diffusion (SD v1-1, v1-4, v1-5, v2-0, v2-1, XL) and modified variants (Taiyi, Safe Latent Diffusion) without any retraining of the watermark model.
  • Image Quality & Invisibility: The method achieves high PSNR (typically >36 dB) and SSIM scores, indicating excellent image quality and invisibility. LPIPS scores are also competitive.
  • Watermark Recoverability: High Normalized Correlation (NC) scores (often >96%) demonstrate robust watermark extraction.
  • Computational Efficiency: MarkPlugger boasts significantly lower FLOPs (1/300th of SOTA methods) and a much smaller parameter count (0.97 million) compared to training-based approaches, making it highly cost-efficient.
  • Robustness: The framework shows good robustness against various attacks, including Gaussian blur, noise, brightness adjustments, cropping, and salt-and-pepper noise. Rotation proved to be a more challenging attack.

Limitations and Future Work

  • Robustness to Rotation: While generally robust, the watermark's performance degrades more significantly under rotation attacks, especially on the Flickr-8K dataset. The authors suggest further investigation into making robustness more generalizable across different datasets and attack types.
  • Parameter Tuning: The paper mentions that parameters like watermark strength ($\alpha$) and loss weight ($\gamma_2$) need careful tuning to balance invisibility and recoverability.
  • Channel Selection: While embedding in any channel works, the paper notes slight performance variations and selects channel 3 as the standard setting. Further research could explore optimal channel selection strategies.
  • Future Work: The authors imply that improving robustness against more severe attacks and exploring more advanced watermark embedding strategies could be future directions.

Summary updated

Abstract

Nowadays, the family of Stable Diffusion (SD) models has gained prominence for its high quality outputs and scalability. This has also raised security concerns on social media, as malicious users can create and disseminate harmful content. Existing approaches involve training components or entire SDs to embed a watermark in generated images for traceability and responsibility attribution. However, in the era of AI-generated content (AIGC), the rapid iteration of SDs renders retraining with watermark models costly. To address this, we propose a training-free plug-and-play watermark framework for SDs. Without modifying any components of SDs, we embed diverse watermarks in the latent space, adapting to the denoising process. Our experimental findings reveal that our method effectively harmonizes image quality and watermark invisibility. Furthermore, it performs robustly under various attacks. We also have validated that our method is generalized to multiple versions of SDs, even without retraining the watermark model.
BibTeX
@misc{zhang2024training,
  author = {Guokai Zhang and Lanjun Wang and Yuting Su and An-An Liu},
  title = {{A Training-Free Plug-and-Play Watermark Framework for Stable Diffusion}},
  year = {2024},
  eprint = {2404.05607},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05607},
}
Figure