Distribution Preserving Watermarking for Large Language Models

20 concepts20 papers1 proposal

As large language models (LLMs) become highly proficient at generating human-like text, identifying the provenance of synthetic content has become a critical safety requirement. Watermarking offers a powerful solution by embedding a hidden statistical signature directly into the generated text.

However, early watermarking techniques suffered from a fundamental flaw: a heavy "quality tax." To embed a signal, these methods typically altered the model's output probabilities, forcing the model to select specific tokens over others. As noted by Zhengmian Hu et al. in their work on Unbiased Watermark for Large Language Models, this distortion creates a direct conflict between the detectability of a watermark and the linguistic quality of the text.

When a watermarking mechanism overrides a model's natural probability distribution, it introduces statistical bias. This bias manifests as repetitive language patterns (Chaoyi Zhu et al.), logic errors (Yidan Wang et al.), and elevated perplexity (Yubing Ren et al.). The model is coerced into choosing less optimal words, making the output sound robotic, awkward, or unnatural (Yuki Takezawa et al.).

To solve this problem, a new paradigm has emerged: distribution-preserving (or unbiased) watermarking. The core insight of this research direction is that it is possible to embed a robust, unerasable signature into text with absolutely zero impact on generation quality.


The Core Insight: Zero-Distortion Sampling

To understand how distribution-preserving watermarking works, we must look at how an LLM selects the next word. At each step, the model outputs a probability distribution (logits) over the entire vocabulary. A standard unwatermarked model samples from this distribution using a source of randomness.

Traditional watermarking alters the probabilities themselves (e.g., boosting a subset of "green" tokens). In contrast, distribution-preserving watermarking leaves the model's probabilities completely untouched. Instead, it changes how the randomness is orchestrated.

Traditional Watermarking:
[Model Logits] ──> [Distort Probabilities (Boost Green Tokens)] ──> [Standard Sampling] ──> Biased Text

Distribution-Preserving Watermarking:
[Model Logits] ───┐
                  ├──> [Coordinated Sampling via Secret Key] ─────────────────────────────> Unbiased Text
[Pseudo-Random] ──┘

As Junfeng Guo et al. demonstrate with DiPmark, if the watermarked text strictly follows the exact statistical distribution of the original model, the text quality remains completely uncompromised. To any observer or perplexity calculator without the secret key, the text is mathematically indistinguishable from natural model output.


Mechanisms of Unbiased Watermarking

Several distinct mathematical frameworks have been developed to achieve this zero-distortion property.

1. Randomized Key Guiding

Instead of using standard, independent random numbers to sample tokens, the authors of Robust Distortion-free Watermarks for Language Models proposed using a randomized watermark key to guide the sampling process. By using the secret key combined with the history of preceding tokens to generate a pseudo-random sequence, the sampling step is subtly coordinated. Because this transformation preserves the marginal probabilities of the tokens, the resulting text remains entirely distortion-free.

2. Maximal Coupling

To ensure that the watermarked distribution matches the original distribution without any statistical drift, Yangxinyu Xie et al. framed the problem through the lens of maximal coupling. In Debiasing Watermarks for Large Language Models via Maximal Coupling, the authors use this mathematical technique to maximize the probability that a sampled token aligns with a target signature, while strictly maintaining the constraint that the marginal distribution matches the original model.

3. Tournament Sampling

For massive, production-scale deployment, Sumanth Dathathri et al. introduced SynthID-Text, a system deployed to millions of users via Google Gemini. Detailed in Scalable watermarking for identifying large language model outputs, this approach uses a "Tournament sampling" algorithm. It selects tokens by evaluating them in pseudo-randomly generated brackets, embedding a robust signature across a sequence of tokens without altering the generation speed or user experience.


Why It Works: The Intuition of Coordinated Randomness

To build an intuition for why this works without degrading quality, imagine rolling a fair six-sided die.

If you want to leave a secret message using a traditional watermarking scheme, you might shave one side of the die to make the number "6" appear 30% of the time instead of the natural 16.7%. Anyone analyzing the frequency of your rolls will quickly notice the die is loaded. In text generation, this "loaded die" is what causes repetitive phrasing and awkward, robotic sentences.

A distribution-preserving watermark leaves the die perfectly fair and unaltered. Every number still has an exact 16.7% chance of appearing. Instead, the watermark is embedded by coordinating the sequence of rolls with a hidden, pseudo-random stream known only to you (the detector).

For example, your secret key might dictate that if a "3" is rolled, the next roll should ideally be a "5". Because you are sampling from a perfectly fair die, a "5" will only appear 16.7% of the time, meaning you cannot force it. However, whenever the die naturally lands on "5" after a "3", you have successfully embedded a fraction of your signal.

To an outsider, a sequence like 3, 5, 1, 6, 2 looks entirely random and perfectly matches the expected distribution of a fair die. But to the detector, the fact that the choices consistently align with the pseudo-random stream over a long enough string of text provides overwhelming statistical proof of AI generation. By shifting the watermark from the probabilities themselves to the coordination of randomness, the quality tax is completely eliminated.


Advanced Frontiers: Capacity and Robustness

With the foundational theory of unbiased watermarking established, recent research focuses on expanding its capabilities along three axes: message capacity, structural adaptation, and robustness.

  • Multi-bit Encoding: Early unbiased watermarks could only provide a binary answer (AI-generated or not). To embed richer metadata like timestamps or user IDs, Xiaoyan Feng et al. developed BiMark, which uses an unbiased multilayer reweighting strategy to embed multi-bit information. Similarly, Ruibo Chen et al. introduced MCMARK in Improved Unbiased Watermark for Large Language Models, partitioning the vocabulary into multiple channels to optimize probability redistribution and boost detectability.
  • Semantic-Level Watermarking: Token-level watermarks can sometimes be disrupted if a user heavily paraphrases the text. To counter this, Jiahao Huo et al. shifted the unit of watermarking from individual tokens to entire sentences in PMark, utilizing multi-channel constraints to prevent distribution distortion while surviving heavy text edits.
  • Entropy-Adaptive Synergy: Not all text segments have the same capacity for watermarking; forcing a watermark when a model is writing code or a highly constrained factual sentence can easily break logic. Yidan Wang et al. designed SymMark to dynamically balance token and semantic entropy. By adaptively combining logits-based and sampling-based watermarking, it ensures signals are only embedded when the model has the linguistic freedom to accommodate them, preserving natural fluency across diverse prompts.

Key Foundational Work

Debiasing Watermarks for Large Language Models via Maximal Coupling (2024) — contributed 1 concept elements to this direction.
Unbiased Watermark for Large Language Models (2024) — contributed 1 concept elements to this direction.
Robust Distortion-free Watermarks for Language Models (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_text_quality_degradation_9422fbff,
  title = {text quality degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/text-quality-degradation/rdr_9422fbffabb04f28b225a55d9b09f8c3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_text_quality_degradation_9422fbff_v1,
  title = {text quality degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/text-quality-degradation/rdr_9422fbffabb04f28b225a55d9b09f8c3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake