Abstract
Standard AI watermarking techniques, which rely on random vocabulary restrictions, often "break" conditional text generation (CTG) tasks like summarization by banning the very words the model needs to use. This paper introduces a Semantic-Aware Watermark (SW) that uses word embeddings to ensure input-related terms stay on the "allowed" list, dramatically improving text quality while maintaining strong detection signals.
Introduction: The Challenge of Watermarking in Conditional Contexts
As AI-generated content becomes indistinguishable from human writing, researchers have turned to "watermarking"—subtly biasing the model's word choices so the resulting text follows a statistical pattern detectable by a secret key.
The current gold standard (Kirchenbauer et al., 2023) works by splitting the vocabulary into "green" (allowed) and "red" (restricted) lists at each step. While this works for creative, open-ended writing, it fails for Conditional Text Generation (CTG). In tasks like summarization or data-to-text generation, the model must use specific entities or facts from the source. If the random algorithm puts a crucial name like "Mandy Patinkin" on the red list, the model is forced to either hallucinate a different name or produce gibberish.
The authors found that applying standard watermarks to CTG can cause performance to plummet by up to 97%, effectively acting as a "hallucination engine."
Methodology: The Semantic-Aware Watermark (SW)
The authors propose a Semantic-Aware Watermark (SW). Instead of purely random partitioning, their approach respects the context of the input through a multi-step process:
- Embedding Similarity: Before generation, the system calculates a similarity matrix ($M$) for the entire vocabulary using word embeddings.
- Contextual "Green-Listing": For a given input, the algorithm identifies the top $k$ semantically related tokens for every word in the source text.
- The Buffer Strategy: These semantically relevant tokens are hard-coded into the "green list." The remaining slots in the green list are then filled randomly (using standard hashing) to maintain the watermark's statistical properties.
- Soft Constraints: They utilize a "soft" watermark approach where a bias ($\delta$) is added to the logits of green-listed tokens, allowing the model some flexibility while still favoring the watermark pattern.
Experimental Evaluation
The researchers evaluated their method across benchmarks including CNN/DailyMail, XSUM, DART, and WebNLG using models like BART-large and Flan-T5.
Performance Recovery
On the WebNLG dataset, the original hard watermark caused a 97% drop in BLEU score. The Semantic-Aware Watermark (SW) recovered this almost entirely, showing a 21.67x improvement in performance over the baseline watermark. In a blind study, human judges preferred SW summaries 55.33% of the time.
Detection Strength and the "Detection Paradox"
Despite the improved text quality, the SW method achieved higher z-scores (a measure of how "obvious" the watermark is to a detector) than the baseline. This indicates very high confidence in AI detection even when the text looks perfectly natural.
However, the paper uncovers a "Detection Paradox": as watermarked text becomes higher quality and more "human-like" (by using tokens similar to the input), it can become harder to distinguish from human writing using standard AUC (Area Under the Curve) metrics. This is because humans also naturally use tokens from the input context.
Conclusion
While SW produces much better text, it results in slightly lower AUC scores for detection compared to the original, more restrictive watermark. This is because the "semantic" part of the green list is somewhat predictable, potentially giving a sophisticated attacker a hint about which tokens are favored. Future work involves finding the perfect sweet spot where the watermark is invisible to humans and robust against attackers, yet doesn't hinder the model's ability to tell the truth.
Summary updated