Evaluating Durability: Benchmark Insights into Multimodal Watermarking

Jielin Qiu, William Han, Xuandong Zhao, Shangbang Long, Christos Faloutsos, Lei LiarXiv 2024 · DOI · arXiv

As generative AI transitions from experimental tools to industrial-scale content engines, the verification of AI-generated media has become a critical challenge. Watermarking—embedding imperceptible identifiers—serves as a primary defense for copyright and provenance. However, the practical utility of these watermarks depends on their "durability" against real-world modifications. A recent study by researchers from Google, Carnegie Mellon, and UCSB introduces a comprehensive benchmark to evaluate the robustness of digital signatures across image and text modalities.

Methodology and System Overview

The research addresses a significant gap in existing literature: the lack of a unified evaluation framework for multimodal distribution shifts. Most watermarking algorithms are tested against isolated distortions, failing to account for the complex transformations content undergoes during distribution, such as social media compression or semantic paraphrasing.

Overview of the multimodal watermarking and detection pipeline under various perturbations.

The authors developed a robust stress test involving:

  1. Watermarking Schemes: Four image methods (e.g., RivaGAN, StegaStamp) and four text methods (e.g., KGW-WM, Blackbox-WM).
  2. Perturbation Library: A massive collection of 100 image perturbations (Noise, Blur, Weather, Digital, Geometric) and 63 text perturbations (Character, Word, and Sentence levels).

Experimental Setup

The benchmark utilized 5,000 image-caption pairs from the COCO validation set. For text-to-image tasks, captions were used to generate images across eight models, including Stable Diffusion and DALLE3. For image-to-text tasks, eight multimodal LLMs (MLLMs) like LLaVA 1.6 and Qwen-VL were employed to generate descriptions. The primary metric for evaluation was Detection Accuracy (Dect Acc), measuring the system's ability to identify watermarks post-perturbation.

Key Insights and Results

The study reveals that while some watermarks are resilient, others are highly susceptible to specific edits. In the image domain, Zoom Blur emerged as a "watermark killer," significantly disrupting the spatial and frequency components relied upon by detection algorithms. Conversely, Glass Blur proved relatively benign.

Performance metrics across various image and text perturbations, highlighting the impact on detection accuracy.

In the text domain, semantic-level changes posed the greatest threat. "Casual" style transfers—rewriting sentences for a different tone—were far more effective at removing watermarks than simple typos or OCR errors. Quantitatively, the drops were stark: some image methods saw accuracy plummet by over 80%, while the Unigram-WM text method experienced a near 99% drop in detection accuracy when faced with sentence-level rewrites.

Conclusion and Future Directions

The findings underscore the fragility of current watermarking standards. While deep-learning-based methods like RivaGAN showed superior stability in images, they still struggle with geometric shifts like rotation. The researchers suggest that future efforts must focus on "distortion-agnostic" watermarking capable of surviving radical semantic and structural changes. While this benchmark provides a vital baseline, extending it to temporal modalities like video and addressing intentional adversarial attacks remain open challenges for the field.

For more details, visit the project website: mmwatermark-robustness.github.io

Summary updated

Abstract

With the development of large models, watermarks are increasingly employed to assert copyright, verify authenticity, or monitor content distribution. As applications become more multimodal, the utility of watermarking techniques becomes even more critical. The effectiveness and reliability of these watermarks largely depend on their robustness to various disturbances. However, the robustness of these watermarks in real-world scenarios, particularly under perturbations and corruption, is not well understood. To highlight the significance of robustness in watermarking techniques, our study evaluated the robustness of watermarked content generated by image and text generation models against common real-world image corruptions and text perturbations. Our results could pave the way for the development of more robust watermarking techniques in the future. Our project website can be found at \url{https://mmwatermark-robustness.github.io/}.
BibTeX
@misc{qiu2024evaluating,
  author = {Jielin Qiu and William Han and Xuandong Zhao and Shangbang Long and Christos Faloutsos and Lei Li},
  title = {{Evaluating Durability: Benchmark Insights into Multimodal Watermarking}},
  year = {2024},
  eprint = {2406.03728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03728},
}
Figure