Benchmarking the Robustness of Image Watermarks

Bang An, Mucong Ding, Tahseen Rabbani, Aakriti Agrawal, Yuancheng Xu, Chenghao Deng, Sicheng Zhu, Abdirisak Mohamed, Yuxin Wen, Tom Goldstein, Furong HuangarXiv 2024 · DOI · arXiv

As generative AI models become increasingly capable of producing photorealistic imagery, the need for reliable provenance mechanisms has never been greater. Digital watermarking—embedding imperceptible signals into images—is a primary candidate for identifying AI-generated content. However, the robustness of these watermarks against intentional or accidental removal remains poorly understood due to a lack of standardized evaluation protocols.

To address this, researchers have developed WAVES (Watermark Analysis via Enhanced Stress-testing), a comprehensive benchmarking toolkit designed to evaluate image watermarking technologies under extreme conditions.

The WAVES Framework

WAVES provides a unified framework for "apple-to-apple" comparisons of watermarking methods. It moves beyond simple image manipulations to include a diverse suite of 26 stress tests categorized into three primary types:

  1. Distortions: Standard geometric (e.g., rotation, cropping) and photometric (e.g., brightness, contrast) transformations that a watermark might encounter in typical web usage.
  2. Regeneration Attacks: These leverage generative models (Diffusion models or VAEs) to "re-generate" an image. This process aims to strip the watermark while preserving the semantic content. WAVES introduces a novel "rinsing" attack—a multi-cycle regeneration process that significantly outperforms standard single-pass methods.
  3. Adversarial Attacks: Targeted attempts to deceive watermark detectors. WAVES introduces "adversarial embedding" attacks, which perturb an image's latent representation to disrupt watermark recovery with minimal visual impact.
The WAVES framework standardizes evaluation across stress tests, metrics, and analysis.

To ensure practical relevance, WAVES utilizes the True Positive Rate (TPR) at a 0.1% False Positive Rate (FPR) as its primary metric. This stringent threshold reflects real-world scenarios where false accusations of AI generation must be minimized.

Experimental Methodology

The benchmark evaluates watermarks across three distinct 5,000-image datasets to ensure generalizability:

  • DiffusionDB: Images generated by Stable Diffusion.
  • MS-COCO: Real-world photographic data.
  • DALL·E 3: Images from OpenAI's proprietary generative model.

By testing across these varied sources, WAVES identifies vulnerabilities that might be specific to certain data distributions or model architectures.

Key Findings and Analysis

The application of WAVES to current state-of-the-art watermarking methods revealed several critical insights:

VAE Vulnerability

A significant discovery is the vulnerability of "latent-space" watermarks, such as Tree-Ring. These methods embed signals within the latent space of a generative model. WAVES demonstrates that if an attacker has access to the underlying Variational Autoencoder (VAE), they can effectively strip the watermark using targeted adversarial perturbations. This highlights a major security risk for systems relying on publicly available VAE architectures.

Resilience of StegaStamp

Surprisingly, StegaStamp—an older, post-processing-based method—exhibited the highest overall robustness. Its resilience is attributed to a training regimen that incorporates a wide variety of simulated real-world distortions, suggesting that diverse data augmentation during training is crucial for watermark durability.

The Power of Regeneration

The benchmark confirms that simple regeneration (adding noise and then denoising) is a highly effective universal attack against "in-processing" watermarks like Stable Signature. The novel "rinsing" attack further amplifies this, stripping watermarks with minimal loss in image quality.

Performance vs. Quality plots illustrate the trade-off between watermark detection (TPR) and image quality (PSNR/CLIP-FID) under various regeneration attacks.

Conclusion

WAVES establishes a new standard for evaluating the robustness of image watermarks. By providing a rigorous, multi-faceted stress-testing environment, it exposes the "Achilles' heels" of current technologies and provides a roadmap for developing more resilient provenance systems. As the arms race between watermark creators and adversaries continues, extensible frameworks like WAVES will be essential for maintaining trust in digital media.

Summary updated

Abstract

In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced, novel variations of diffusive, and adversarial attacks. Our evaluation examines two pivotal dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. Our novel, comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarks. The project is available at https://wavesbench.github.io/
BibTeX
@misc{an2024benchmarking,
  author = {Bang An and Mucong Ding and Tahseen Rabbani and Aakriti Agrawal and Yuancheng Xu and Chenghao Deng and Sicheng Zhu and Abdirisak Mohamed and Yuxin Wen and Tom Goldstein and Furong Huang},
  title = {{Benchmarking the Robustness of Image Watermarks}},
  year = {2024},
  eprint = {2401.08573},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.08573},
}
Figure