Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedy

Yu Fu, Deyi Xiong, Yue Dong

Abstract

Standard AI watermarking techniques often degrade the quality of conditional text generation (CTG) tasks, such as summarization, by randomly penalizing essential source-derived keywords. This paper identifies a "Detection Paradox" and proposes a Semantic-Aware Watermark (SW) to mitigate performance loss. By protecting semantically relevant tokens from the input context, SW recovers model performance—achieving up to a 21x improvement in BLEU scores over baselines—while maintaining a robust detection signal.

Introduction

As Large Language Models (LLMs) become integrated into information workflows, the ability to distinguish between human and AI-generated text is paramount. Watermarking, which biases token selection toward a pseudo-random "green list," has emerged as a leading detection strategy. However, in Conditional Text Generation (CTG), where outputs must strictly adhere to source inputs, standard watermarking (OW) often places critical source tokens on the "red list." This forces models to either omit necessary information or hallucinate alternatives, leading to performance drops as high as 97%.

Proposed Method: Semantic-Aware Watermark (SW)

The authors introduce the Semantic-Aware Watermark (SW) to align the watermarking process with the semantic requirements of CTG. The method involves:

  1. Semantic Mapping: Identifying tokens semantically related to the input sequence using word embeddings.
  2. Green List Protection: Pre-approving these relevant tokens by ensuring they are always in the green list, regardless of the hash-based partition.
  3. Soft Logit Bias: Applying a bias $\delta$ to green list tokens to encourage their selection without strictly forbidding others.
  4. Breadth Control ($k$): A hyperparameter $k$ determines the number of semantic neighbors protected for each input token.

Experimental Evaluation

The method was evaluated on summarization (CNN/DailyMail, XSUM) and data-to-text (DART, WebNLG) benchmarks using BART and Flan-T5 models to ensure generalizability across architectures.

Results and Analysis

Quality Recovery

The primary advantage of SW is its ability to maintain text quality even under "hard" watermarking constraints. On the DART dataset, while OW caused BLEU scores to plummet from 47.78 to 6.65, SW maintained a score of 41.04.

BLEU scores for SW vs OW across varying watermark strengths (delta) and green list sizes (gamma). SW consistently maintains higher quality.

The Detection Paradox and Signal Strength

A key finding is the "Detection Paradox": because humans naturally use source-relevant tokens, protecting these tokens in the watermark makes AI text appear more "human-like" to simple classifiers (lower AUC), even though the statistical evidence of the watermark is stronger.

Z-score comparison showing that SW provides a more robust detection signal (higher z-score) than the original watermark as the bias delta increases.

Despite the paradox, SW yields significantly higher z-scores than OW, indicating a more detectable signal for specialized tools. Human evaluations on XSUM also favored SW-generated text 55.33% of the time over OW, effectively eliminating the "watermark tax" on quality.

Conclusion

Semantic-Aware Watermarking effectively addresses the quality degradation inherent in standard watermarking for CTG tasks. While the overlap with natural human word choice presents new challenges for binary classification, the increased statistical signal strength (z-score) offers a path forward for reliable AI detection. Future research will focus on balancing semantic relevance with detection accuracy through automated hyperparameter tuning.

Summary updated

BibTeX
@inproceedings{fu2024watermarking,
  author = {Yu Fu and Deyi Xiong and Yue Dong},
  title = {{Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedy}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29756},
}
Figure