Watermarking Diffusion Language Models

Thibaud Gloaguen, Robin Staab, Nikola Jovanovic, Martin T. Vechev

Introduction

As Large Language Models (LLMs) become ubiquitous, the ability to distinguish AI-generated text from human writing is critical for safety and accountability. Most current watermarking techniques, such as the popular "Red-Green" scheme, are designed for Autoregressive Language Models (ARLMs). These models generate text sequentially from left to right, allowing them to hash previous tokens to determine the bias for the next token.

However, Diffusion Language Models (DLMs) generate text iteratively and often in a non-sequential order (e.g., middle-out). Because the "previous context" is frequently missing or incomplete during generation, standard watermarking hashes fail. A naive approach—only watermarking when full context is available—results in a signal too weak for reliable detection. Researchers from ETH Zurich have addressed this by developing the first specialized watermarking scheme for DLMs, framing the task as a constrained optimization problem.

Methodology: Optimization-Based Watermarking

The proposed framework treats watermarking as an optimization task: maximize the number of "green" tokens in the final sequence while minimizing the KL divergence from the model's original probability distribution.

Comparison of watermarking schemes for Autoregressive vs. Diffusion models.

The solution introduces two key components to the generation process:

  1. Expectation Boost: Instead of waiting for a fixed context, the model applies the watermark based on the expected context. It evaluates the current probability distribution of surrounding masked tokens to calculate the likelihood of various hashes.
  2. Predictive Bias: The model doesn't just favor making the current token "green"; it also favors tokens that will serve as "good context" for future tokens, facilitating their own "green" classification later in the process.

Crucially, while the generation process is sophisticated, the detector remains a simple, standard binomial test. This ensures the watermark can be detected using existing infrastructure designed for autoregressive models.

Experimental Evaluation

The researchers evaluated their method using several benchmarks:

  • WaterBench: A comprehensive benchmark for LLM watermarking, utilizing long-answer tasks from ELI-5, Finance-QA, and Alpaca-Farm.
  • C4 (Realnewslike): Used specifically to test infilling tasks.
  • Standard Benchmarks: MMLU, HellaSwag, ARC-C, GPQA, MBPP, and GSM8K were used to ensure the watermark did not degrade model intelligence.

Key Results and Insights

The scheme achieves a True Positive Rate (TPR) of >99% at a 1% False Positive Rate (FPR). Crucially, text quality remains high. For example, on the LLADA-8B model, average benchmark accuracy only decreased from 59.4% to 56.0% while achieving near-perfect detection. Despite the additional mathematical overhead, the computational cost is minimal, adding only about 9ms per token.

Performance comparison showing TPR vs. Perplexity (left) and Number of Tokens (right).

Key Insights:

  • Proactive Generation: Unlike reactive AR watermarks, this method is "future-proof" by using Predictive Bias to seed the sequence for high detectability.
  • Order Independence: The method is effective regardless of whether the DLM unmasks tokens left-to-right, right-to-left, or randomly.
  • Robustness: The watermark remains detectable even after 30% of the text is modified through deletion or substitution and is resilient to paraphrasing.

Limitations and Future Work

The watermark's effectiveness is sensitive to sampling temperature; at very low temperatures, there is less flexibility to shift probabilities without impacting quality. Additionally, while larger context windows mitigate "scrubbing" attacks, extremely small windows remain a challenge. Future research may extend this optimization-based approach to multi-modal diffusion models for images and audio.

Summary updated

BibTeX
@inproceedings{gloaguen2026watermarking,
  author = {Thibaud Gloaguen and Robin Staab and Nikola Jovanovic and Martin T. Vechev},
  title = {{Watermarking Diffusion Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=3aBWTYGcaT},
}
Figure