TL;DR
This paper demonstrates that language models can directly learn to generate watermarked text through their weights via a process called watermark distillation, removing the need for specialized decoding algorithms. While this opens the door to watermarking open-source models, it also reveals a severe security flaw: adversaries can easily "spoof" a model's watermark to falsely attribute malicious or harmful text to a victim provider.
What Problem Does This Paper Address?
As language models (LMs) grow more pervasive, distinguishing human text from machine-generated text is crucial for combating disinformation and academic dishonesty. Currently, the standard way to watermark an LM is decoding-based watermarking, which alters the model's token selection probabilities at inference time using a secret key.
However, this approach has two major flaws:
- Incompatibility with Open Models: If a provider releases a model's weights, a user can simply bypass the specialized watermarked decoder and use standard decoding to generate unwatermarked text.
- Vulnerability to Spoofing: If watermarks are used to prove the provenance (origin) of a text, an attacker might try to generate harmful text that carries a victim model's watermark, destroying the victim's reputation.
The authors ask a fundamental question: Can an LM directly learn to generate watermarked text under its natural sampling distribution, using just its weights?
Methods
To study the learnability of watermarks, the authors introduce watermark distillation, where a student model is trained to mimic a teacher model that uses decoding-based watermarking. They propose two distinct flavors:
- Logit-based Watermark Distillation: The student model is trained to minimize the KL divergence between its next-token probability distribution and the modified distribution produced by the watermarked teacher model. This requires access to the teacher's logits and a shared tokenizer.
- Sampling-based Watermark Distillation: The teacher model generates a dataset of watermarked text completions. The student model is then fine-tuned on this dataset using standard cross-entropy loss. This method is highly flexible because it only requires text samples, meaning the student and teacher can have entirely different architectures or tokenizers.
The authors evaluate these methods across three prominent decoding-based watermarking schemes: KGW (green-list bias), Aar (boosting continuous hash scores), and KTH (robust sequence alignment).
Datasets
The experiments utilize several standard datasets for training and evaluation:
- OpenWebText: Used as the text corpus for both logit-based distillation training and generating prompts for sampling-based distillation.
- C4 (RealNewsLike subset): Used to provide evaluation prompts for testing watermark detectability and generation quality.
- Wikipedia Articles & arXiv Papers: Used to evaluate how well the distilled watermarks generalize across different text domains.
- HarmfulQ: A benchmark of 200 toxic questions used to evaluate the real-world impact of a watermark spoofing attack.
Key Insights and Ideas
- Weights-Based Watermarks are Highly Learnable: LMs can successfully internalize complex, hash-based watermarking rules directly into their parameters. Once learned, they naturally output watermarked text using standard decoding strategies.
- The Distortion Trade-off: Watermarks that cause higher distortion (i.e., alter the text more drastically from natural human language) are significantly easier and more sample-efficient for the student to learn than subtle, low-distortion watermarks.
- Watermarks Cannot Prove Provenance: Because sampling-based distillation only requires text outputs, an adversary can easily steal a victim model's watermark signal. This means watermarks should only be used to detect if text is AI-generated, not which specific model generated it.
- Fragility to Fine-Tuning: Weights-based watermarks are easily "washed out" if the model is subsequently fine-tuned on normal, unwatermarked text.
Key Results
The authors successfully distilled watermarks into Llama 2 7B and Pythia 1.4B models.
As shown in the main results table, both distillation methods achieve exceptionally low detection p-values and near-perfect AUROC scores for high-distortion settings:
Furthermore, weights-based watermarking proved highly robust to changes in decoding parameters at inference time, such as varying nucleus sampling thresholds ($p$) or temperatures ($t$):
The Spoofing Attack in Action
To demonstrate the danger of spoofing, the authors trained an adversary model (Alpaca-7B) using samples from a safe, watermarked victim model (Llama 2-Chat 7B). While the victim model successfully refused 100% of harmful prompts from the HarmfulQ benchmark, the distilled adversary model answered 71% of the harmful prompts. Crucially, the adversary's toxic responses retained the victim's watermark with a median p-value of 0.002, meaning the harmful text would be falsely blamed on the victim.
Limitations and Future Work
The primary limitation of weights-based watermarking is its lack of robustness to continued fine-tuning. If an open-source user fine-tunes a watermarked model on standard text, the watermark capability disappears within a few hundred steps.
Additionally, lower-distortion watermarks exhibit high sample complexity, requiring massive amounts of data to learn effectively. The authors point out that developing techniques to make weights-based watermarks robust against fine-tuning and exploring robust defenses against watermark spoofing remain critical open challenges for future research.
Summary updated