Digital watermarking for Large Language Models (LLMs) operates by subtly biasing the model's word choices. By slightly increasing the probability of certain "green" tokens and decreasing others, researchers can embed a statistical signature that is invisible to humans but detectable by algorithms. However, this signature is not a single stamp; it is a distribution spread across the entire text. As a result, the reliability of detection is fundamentally tied to the length of the generated output.
The Sample Size Bottleneck
The core challenge of watermarking is a classic problem of statistical power. As John Kirchenbauer et al. explain in On the Reliability of Watermarks for Large Language Models, a watermark is essentially a "bias" that becomes detectable only when observed over many samples. If you flip a slightly weighted coin five times, you cannot reliably tell if it is biased; if you flip it five hundred times, the deviation from the expected 50/50 split becomes undeniable.
In the context of LLMs, each token is a "coin flip." In long-form articles, the cumulative bias is easy to spot. But for short-form content like tweets, social media posts, or brief emails, the number of tokens is often insufficient to distinguish the watermark's bias from natural human language patterns. Shangqing Tu et al. note in WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models that this creates a low signal-to-noise ratio, where the "noise" of natural linguistic variation drowns out the "signal" of the watermark.
The Information Density Problem
The difficulty is not just about the number of words, but the "entropy" or information density of those words. Zhuohao Yu et al. argue in SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders that extremely short outputs, such as single-word responses, lack the semantic variance necessary to host a signature without fundamentally altering the intended meaning.
This bottleneck appears across different domains of watermarking:
- Network Forensics: Alfonso Iacovazzi et al. show in DropWat: An Invisible Network Flow Watermark for Data Exfiltration Traceback that packet-drop watermarks require a minimum traffic volume, making them ineffective for short-lived interactive sessions.
- Image Forensics: Even in visual media, Hai Ci et al. demonstrate in Can Simple Averaging Defeat Modern Watermarks? that extracting certain additive patterns requires a sample size of 50 to 100 instances to isolate the signal from the underlying content.
- Benchmark Contamination: When trying to prove an LLM was trained on a specific dataset, Tom Sander et al. found that brief answers or labels provide insufficient "radioactive" traces for reliable detection in Detecting Benchmark Contamination Through Watermarking.
Breaking the Length Barrier
To address this sensitivity to token length, recent research has focused on increasing the "density" of the watermark signal. The goal is to reach high-confidence detection thresholds with fewer tokens.
One approach, proposed by Chaoyi Zhu et al. in Duwak: Dual Watermarks in Large Language Models, uses a dual-layer scheme that embeds patterns into both the token probability distribution and the sampling process. By combining these independent signals, the authors achieved reliable detection using 70% fewer tokens than standard methods.
Another strategy involves changing how we look for the signal. Traditional methods often rely on simple frequency counts (how many "green" tokens appear). However, Xingchi Li et al. show in A Likelihood Based Approach for Watermark Detection that leveraging the model's own Next Token Probabilities (NTPs) can significantly boost statistical power. Their likelihood-based framework enables reliable provenance tracking for sequences as short as eight words, effectively pushing the detection threshold closer to the scale of individual sentences.
Key Foundational Work
DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models (2023) — contributed 1 concept elements to this direction.
DropWat: An Invisible Network Flow Watermark for Data Exfiltration Traceback (2018) — contributed 1 concept elements to this direction.