On the Learnability of Watermarks for Language Models

Chenchen Gu, Xiang Lisa Li, Percy Liang, Tatsunori Hashimoto

TL;DR

This paper demonstrates that language models can directly learn to generate watermarked text through their weights via a process called watermark distillation, removing the need for specialized decoding algorithms. While this opens the door to watermarking open-source models, it also reveals a severe security flaw: adversaries can easily "spoof" a model's watermark to falsely attribute malicious or harmful text to a victim provider.

What Problem Does This Paper Address?

As language models (LMs) grow more pervasive, distinguishing human text from machine-generated text is crucial for combating disinformation and academic dishonesty. Currently, the standard way to watermark an LM is decoding-based watermarking, which alters the model's token selection probabilities at inference time using a secret key.

However, this approach has two major flaws:

  1. Incompatibility with Open Models: If a provider releases a model's weights, a user can simply bypass the specialized watermarked decoder and use standard decoding to generate unwatermarked text.
  2. Vulnerability to Spoofing: If watermarks are used to prove the provenance (origin) of a text, an attacker might try to generate harmful text that carries a victim model's watermark, destroying the victim's reputation.

The authors ask a fundamental question: Can an LM directly learn to generate watermarked text under its natural sampling distribution, using just its weights?

Overview of decoding-based vs. weights-based watermarking

Methods

To study the learnability of watermarks, the authors introduce watermark distillation, where a student model is trained to mimic a teacher model that uses decoding-based watermarking. They propose two distinct flavors:

  • Logit-based Watermark Distillation: The student model is trained to minimize the KL divergence between its next-token probability distribution and the modified distribution produced by the watermarked teacher model. This requires access to the teacher's logits and a shared tokenizer.
  • Sampling-based Watermark Distillation: The teacher model generates a dataset of watermarked text completions. The student model is then fine-tuned on this dataset using standard cross-entropy loss. This method is highly flexible because it only requires text samples, meaning the student and teacher can have entirely different architectures or tokenizers.

The authors evaluate these methods across three prominent decoding-based watermarking schemes: KGW (green-list bias), Aar (boosting continuous hash scores), and KTH (robust sequence alignment).

Datasets

The experiments utilize several standard datasets for training and evaluation:

  • OpenWebText: Used as the text corpus for both logit-based distillation training and generating prompts for sampling-based distillation.
  • C4 (RealNewsLike subset): Used to provide evaluation prompts for testing watermark detectability and generation quality.
  • Wikipedia Articles & arXiv Papers: Used to evaluate how well the distilled watermarks generalize across different text domains.
  • HarmfulQ: A benchmark of 200 toxic questions used to evaluate the real-world impact of a watermark spoofing attack.

Key Insights and Ideas

  • Weights-Based Watermarks are Highly Learnable: LMs can successfully internalize complex, hash-based watermarking rules directly into their parameters. Once learned, they naturally output watermarked text using standard decoding strategies.
  • The Distortion Trade-off: Watermarks that cause higher distortion (i.e., alter the text more drastically from natural human language) are significantly easier and more sample-efficient for the student to learn than subtle, low-distortion watermarks.
  • Watermarks Cannot Prove Provenance: Because sampling-based distillation only requires text outputs, an adversary can easily steal a victim model's watermark signal. This means watermarks should only be used to detect if text is AI-generated, not which specific model generated it.
  • Fragility to Fine-Tuning: Weights-based watermarks are easily "washed out" if the model is subsequently fine-tuned on normal, unwatermarked text.

Key Results

The authors successfully distilled watermarks into Llama 2 7B and Pythia 1.4B models.

As shown in the main results table, both distillation methods achieve exceptionally low detection p-values and near-perfect AUROC scores for high-distortion settings:

Main results across different watermarking strategies

Furthermore, weights-based watermarking proved highly robust to changes in decoding parameters at inference time, such as varying nucleus sampling thresholds ($p$) or temperatures ($t$):

Robustness of distilled watermarks to decoding parameters

The Spoofing Attack in Action

To demonstrate the danger of spoofing, the authors trained an adversary model (Alpaca-7B) using samples from a safe, watermarked victim model (Llama 2-Chat 7B). While the victim model successfully refused 100% of harmful prompts from the HarmfulQ benchmark, the distilled adversary model answered 71% of the harmful prompts. Crucially, the adversary's toxic responses retained the victim's watermark with a median p-value of 0.002, meaning the harmful text would be falsely blamed on the victim.

Limitations and Future Work

The primary limitation of weights-based watermarking is its lack of robustness to continued fine-tuning. If an open-source user fine-tunes a watermarked model on standard text, the watermark capability disappears within a few hundred steps.

Additionally, lower-distortion watermarks exhibit high sample complexity, requiring massive amounts of data to learn effectively. The authors point out that developing techniques to make weights-based watermarks robust against fine-tuning and exploring robust defenses against watermark spoofing remain critical open challenges for future research.

Summary updated

Abstract

Watermarking of language model outputs enables statistical detection of model-generated text, which can mitigate harms and misuses of language models. Existing watermarking strategies operate by altering the decoder of an existing language model. In this paper, we ask whether language models can directly learn to generate watermarked text, which would have significant implications for the real-world deployment of watermarks. First, learned watermarks could be used to build open models that naturally generate watermarked text, enabling watermarking for open models, where users can control the decoding procedure. Second, if watermarking is used to determine the provenance of generated text, an adversary can hurt the reputation of a victim model by spoofing its watermark and generating damaging watermarked text. To investigate the learnability of watermarks, we propose watermark distillation, which trains a student model to behave like a teacher model that uses decoding-based watermarking. We test our approach on three decoding-based watermarking strategies and various hyperparameter settings, finding that models can learn to generate watermarked text with high detectability. We also find limitations to learnability, including the loss of watermarking capabilities under fine-tuning on normal text and high sample complexity when learning low-distortion watermarks.
BibTeX
@inproceedings{gu2024learnability,
  author = {Chenchen Gu and Xiang Lisa Li and Percy Liang and Tatsunori Hashimoto},
  title = {{On the Learnability of Watermarks for Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=9k0krNzvlV},
}
Figure