A Private Watermark for Large Language Models

Aiwei Liu, Leyi Pan, Xuming Hu, Shu'ang Li, Lijie Wen, Irwin King, Philip S. Yu

Introduction

As Large Language Models (LLMs) become integrated into daily information consumption, the ability to distinguish machine-generated text from human writing is essential for maintaining information integrity and protecting intellectual property. Traditional watermarking techniques embed signals by subtly biasing token selection. However, these methods typically rely on a "shared secret" key. If this key is made public for verification purposes, it can be exploited by malicious actors to forge the watermark, falsely attributing synthetic content to a trusted source.

This paper introduces UPV (Unforgeable Publicly Verifiable), a novel watermarking system that resolves this tension. By employing a pair of distinct neural networks—a private one for embedding and a public one for detection—UPV enables universal verification without compromising the security of the generation process.

Methodology: The UPV Framework

The core innovation of UPV lies in its computational asymmetry, drawing inspiration from one-way functions in cryptography. The system consists of two primary components:

  1. The Private Generator: A small Feed-Forward Network (FFN) that analyzes a local window of preceding tokens to determine a "green list" for the next token. The LLM's logits are then biased to favor these tokens.
  2. The Public Detector: An LSTM-based network trained to identify the watermark patterns. Crucially, while the detector can recognize the watermark, reverse-engineering the generator's specific rules from the detector is computationally prohibitive.

To enhance efficiency, both networks share the same token embedding parameters. This allows the detector to achieve high accuracy with a minimal parameter footprint (~43k parameters). Additionally, the system utilizes "cyclic labeling" to prevent attackers from probing the detector to deduce generation rules.

Overview of the UPV system architecture, showing the private generator and public detector.

Experimental Setup

The researchers evaluated UPV using models of varying scales, including GPT-2, OPT-1.3B, and LLaMA-7B. Testing was conducted on the C4 (Colossal Clean Crawled Corpus) for general performance and the DBPEDIA CLASS dataset to assess domain robustness. Performance was benchmarked against standard key-based watermarking methods.

Results and Discussion

UPV demonstrates exceptional performance across several critical metrics:

  • Detection Accuracy: The system achieves a near-perfect F1 score of 99%, matching the effectiveness of traditional key-based methods.
  • Unforgeability: "Reverse training" attacks, where an adversary attempts to train a surrogate generator using the public detector, consistently failed to produce high-quality forgeries.
  • Text Quality: The impact on text naturalness is negligible. Metrics such as perplexity (PPL) and BLEU scores show only marginal changes, ensuring the watermark does not degrade the LLM's utility.
  • Robustness: The watermark remains detectable even after text is rewritten by other models like GPT-3.5, often outperforming existing techniques in these scenarios.
Performance metrics showing detection accuracy, unforgeability, and impact on text perplexity.

Limitations

While highly effective, the system's security is sensitive to the token window size ($w$). Smaller windows may be susceptible to frequency analysis. Furthermore, while robust against single rewrites, the system's resilience against iterative adversarial attacks or sophisticated "copy-paste" strategies requires further investigation.

Conclusion

UPV represents a significant step forward in LLM transparency. By decoupling the generation and detection mechanisms, it provides a practical path toward public verification of AI-generated content. This "unforgeable" approach ensures that while anyone can verify the origin of a text, the power to create a valid watermark remains securely in the hands of the model provider.

Summary updated

BibTeX
@inproceedings{liu2024private,
  author = {Aiwei Liu and Leyi Pan and Xuming Hu and Shu'ang Li and Lijie Wen and Irwin King and Philip S. Yu},
  title = {{A Private Watermark for Large Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=gMLQwKDY3N},
}
Figure