Lost in Overlap: Exploring Logit-based Watermark Collision in LLMs

Yiyang Luo, Ke Lin, Chao Gu, Jiahui Hou, Lijie Wen, Luo PingNAACL 2025 · DOI

Introduction

As Large Language Models (LLMs) become ubiquitous, "watermarking" their output has emerged as the primary defense for copyright and provenance. This technique subtly shifts the probability distribution (logits) of tokens during generation to embed a statistical "signature" detectable by algorithms but invisible to humans. However, a critical vulnerability exists: watermark collision.

Conceptual overview of watermark collision where sequential watermarking by Alice and Bob leads to detection failure for both.

Watermark collision occurs when applying a second watermark—often via downstream tasks like paraphrasing or translation—effectively "erases" the first one. This happens because the underlying probability distributions become tangled, providing a potent new strategy for watermark attacks.

The Problem: A "Wild West" of Watermarks

The authors identify a looming scenario where text is processed by multiple LLMs, each attempting to leave its own mark. This isn't merely an accidental byproduct of multi-model workflows; it is a fundamental weakness that can be exploited as a "collision attack" to strip away existing protections without destroying the text's readability.

Methodology: The Collision Pipeline

The research formalizes a "Collision Pipeline" involving three primary actors:

  1. The Watermarker ($W$): The original LLM generating watermarked text.
  2. The Collider ($C$): An attacking LLM that performs a task (e.g., paraphrasing) while simultaneously embedding its own watermark.
  3. The Detectors: Algorithms designed to identify the specific signatures of $W$ or $C$.
The experimental pipeline for evaluating watermark collision across paraphrasing, back-translation, and mask-and-fill tasks.

Detection typically relies on a "null hypothesis"—the assumption that natural text follows a random distribution of "green" and "red" tokens. When a second watermark is applied, it creates an entangled distribution. Mathematically, this new distribution no longer conforms to the original watermark’s rules, causing the original detector to fail. The study evaluated three types of colliders: paraphrasers, back-translators, and mask-and-fill models.

Experimental Setup

The study utilized the C4 (Colossal Clean Crawled Corpus) dataset for text generation. Evaluation involved several prominent open-source LLMs, including LLaMA-2-13B, Qwen2-7B, OPT-1.3B, and RoBERTa-LARGE. These were tested against three major watermarking algorithms: KGW (Kirchenbauer et al.), PRW (Zhao et al.), and SIR (Liu et al.).

Key Insights and Results

The results demonstrate that collision is a highly effective and stealthy attack strategy.

  • The "Tug-of-War" Effect: Watermarks are competitive rather than additive. When two watermarks overlap, they compete for the "probability budget" of the text, with one usually emerging stronger at the significant expense of the other.
  • Detection Collapse: In many scenarios, the True Positive Rate (TPR) of the original watermark dropped from near 100% to below 40% after a single collision with a strong paraphraser.
  • Quality Preservation: Unlike traditional "heavy" attacks that scramble text, collision attacks preserve fluency. Perplexity scores remained stable, and in some cases, the double-watermarked text actually showed improved perplexity.
  • Multi-round Erasure: Running text through a paraphraser multiple times with different keys acts like a digital sandblaster, progressively eroding the original watermark until it is completely undetectable.

Limitations and Future Outlook

While the study highlights a universal vulnerability across different model architectures, it focused primarily on paraphrasing, translation, and mask-and-fill. Future research is needed to explore how collision behaves in complex tasks like long-form summarization. Currently, there is a "defense gap," as no "collision-proof" watermark yet exists. Developing robust watermarking schemes that can survive or even signal the presence of an overlapping signature remains a critical area for future work.

Summary updated

BibTeX
@inproceedings{luo2025lost,
  author = {Yiyang Luo and Ke Lin and Chao Gu and Jiahui Hou and Lijie Wen and Luo Ping},
  title = {{Lost in Overlap: Exploring Logit-based Watermark Collision in LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.37},
}
Figure