SAT-LDM: Provably Generalizable Image Watermarking for Latent Diffusion Models with Self-Augmented Training

Lu Zhang, Liang ZengSubmission to ICLR 2025 · Rejected

TL;DR

Current diffusion-native watermarking techniques often inject noticeable visual artifacts because they train VAE decoders on biased external datasets rather than the model's actual generative pipeline. SAT-LDM solves this by using "self-augmented training"—generating latent representations directly from the diffusion model using empty prompts ("free generation")—completely eliminating external image datasets. Backed by a tight theoretical generalization bound, SAT-LDM dramatically cuts image distortion (slashing FID by over 50%) while keeping watermark detection accuracy above 96% across varied styles and attacks.

What Problem Does This Paper Address?

Embedding invisible, robust watermarks into AI-generated images is crucial for copyright provenance and detecting deepfakes. Recent diffusion-native approaches (like Stable Signature and FSW) fine-tune the Latent Diffusion Model's (LDM) VAE decoder to bake watermarks directly into the generation process.

However, these methods suffer from an inherent training–testing mismatch:

  • During training: Real images pass through the VAE encoder to obtain latents, which are then passed to the decoder (image → VAE encoder → VAE decoder).
  • During inference: Text prompts pass through the diffusion UNet denoiser to yield latents, which the decoder converts to pixels (prompt → UNet → VAE decoder).

Relying on external image datasets for training causes the watermarked decoder to overfit to specific dataset distributions. When deployed on unseen prompts in the wild, the model often produces visible discoloration, glare, or blurriness.

Methods

SAT-LDM eliminates external training data and aligns the training pipeline directly with inference:

  1. Self-Augmented Training (Free Generation Distribution): Instead of encoding external images, SAT-LDM feeds empty prompts ("") through the frozen UNet denoiser: $z = U(\text{""}, \epsilon)$. Because unconditional generation serves as a natural proxy for the average of all prompt-conditional distributions, the latent representations match the distribution the decoder encounters during real-world inference.
  2. Modular Decoder Plugin: Rather than altering the underlying VAE decoder weights directly, SAT-LDM introduces a lightweight message processor that plugs into the decoder's intermediate features. This preserves the base model's integrity and makes the watermark module easily swappable.
  3. Perspective-Resilient Extractor: A Spatial Transformer Network (STN) is placed ahead of the message extraction network ($T_m$), significantly boosting robustness against camera perspective distortions (e.g., photographing a printout).
  4. Theoretical Generalization Guarantee: The authors derive a generalization bound based on the first Wasserstein distance ($W_1$) between the training and test latent distributions, mathematically proving that training on the free distribution yields a tighter error bound.

Datasets

  • Training: Completely data-free. SAT-LDM trains on 30K self-generated latents from the base diffusion model (Stable Diffusion v1.5 / v2.1). (A 30K subset of LAION-400M is used only as an external-data baseline).
  • Evaluation:
  • Standard prompt sets: COCO, LAION-400M, and Diffusion Prompts (1,000 prompts each).
  • AI-Generated Prompts Benchmark: A newly introduced evaluation benchmark consisting of 1,000 diverse prompts synthesized via GPT-4 across 10 distinct categories (e.g., natural landscapes, traditional art styles, modern/abstract art, fantasy/sci-fi, and everyday objects) to rigorously stress-test style generalization.

Key Insights and Ideas

  • Unconditional generation mirrors the prompt space: In theory, averaging latent distributions across all possible prompts yields the unconditional distribution $U_\sharp(\mu_p \times \mu_\epsilon) \approx U_\sharp \mu_\epsilon$. Thus, training on "empty-prompt" generations acts as an ideal neutral prior across virtually any text prompt.
  • Smaller Wasserstein distance implies higher fidelity: Empirical measurements show the Wasserstein distance ($W_1$) between the training latents and test latents drops from 911.4 (external data) to 504.4 (free generation). This directly corresponds to fewer visual artifacts in practice.
  • Privacy and copyright compliance by default: Because no external images or scraped text-image pairs are used during training, developers can watermark diffusion pipelines without risking intellectual property or data privacy infringements.

Key Results

On the 1K AI-Generated Prompts evaluation using Stable Diffusion v1.5:

  • Image Quality: SAT-LDM achieves 40.58 dB PSNR and a 2.40 FID, dramatically outperforming FSW (27.32 dB / 18.92 FID) and Stable Signature (29.17 dB / 9.49 FID). The low FID indicates watermarked outputs are virtually indistinguishable from non-watermarked ones.
  • Robustness Under Attacks: Evaluated against 7 attack types (Gaussian blur, Gaussian noise, brightness, contrast, desaturation, perspective warp, and JPEG compression), SAT-LDM maintains a 0.988 bit accuracy under adversarial attacks and 1.000 without attacks (for 100-bit payloads).
  • Sampling & Guidance Invariance: SAT-LDM generalizes seamlessly across sampling algorithms (DDPM, DDIM, LMS, Euler) and maintains stability across classifier-free guidance scales spanning from 2 up to 18.

Limitations and Future Work

  • High Guidance Discrepancy: The core assumption—that the free generation distribution matches the prompt-conditioned distribution—weakens at extreme guidance scales ($gs \ge 14$), where strong guidance pulls latents away from unconditional space and slightly degrades the Pareto frontier between quality and robustness.
  • Prompt Benchmarking Biases: While the 1,000 GPT-4 evaluation prompts span 10 diverse genres, they may still carry subtle language model stylistic biases.
  • Future Directions: The authors suggest exploring dataset distillation to reduce the required 30K free generation training samples even further, potentially enabling near-instant watermarking adaptation.

Summary updated

Abstract

The proliferation of AI-generated images necessitates effective watermarking to protect intellectual property and identify fake content. While existing training-based watermarking methods show promise, they often struggle with generalization across diverse image styles and tend to produce noticeable artifacts. To this end, we introduce a provably generalizable image watermarking method for Latent Diffusion Models with Self-Augmented Training (SAT-LDM), which aligns the training and testing phases by a free generation distribution to bolster the watermarking module’s generalization capabilities. We theoretically consolidate our method by proving that the free generation distribution contributes to its tight generalization bound without the need to collect new data. Extensive experimental results demonstrate that SAT-LDM achieves robust watermarking while significantly improving the quality of watermarked images across diverse styles. Furthermore, we conduct experimental analyses to demonstrate the strong generalization abilities of SAT-LDM. We hope our method offers a practical and convenient solution for securing high-fidelity AI-generated content.
BibTeX
@misc{zhang2025sat,
  author = {Lu Zhang and Liang Zeng},
  title = {{SAT-LDM: Provably Generalizable Image Watermarking for Latent Diffusion Models with Self-Augmented Training}},
  year = {2025},
  eprint = {2501.00463},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.00463},
}
Figure