SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models

Amirhossein Dabiriaghdam, Lele WangEMNLP 2025

TL;DR

This paper introduces SimMark, a novel algorithm for watermarking text generated by Large Language Models (LLMs). SimMark embeds a statistical signal by analyzing the similarity of sentence embeddings, making LLM outputs traceable without needing access to the model's internal workings. This approach offers robust detection against paraphrasing attacks while maintaining text quality, making it a practical solution for various LLM applications.

What Problem Does This Paper Address?

The widespread adoption of LLMs like GPT-4 has enabled the generation of human-like text at an unprecedented scale. While this offers numerous benefits, it also introduces significant risks, including academic plagiarism, disinformation campaigns, and the erosion of accountability. Reliably detecting AI-generated content is crucial for addressing these challenges, but LLMs are designed to emulate human writing styles, making their outputs difficult to distinguish from human-authored text. Existing watermarking techniques often struggle with robustness against paraphrasing attacks or require access to model internals, limiting their applicability. This paper tackles the challenge of creating a robust, practical, and broadly applicable watermarking method for LLM-generated text.

Methods

SimMark is a sentence-level semantic watermarking algorithm. Its core idea is to leverage the similarity of semantic sentence embeddings.

Watermarked Text Generation: 1. Embedding: Sentences are embedded using a pre-trained, general-purpose semantic embedding model (e.g., Instructor-Large). Crucially, the model is not fine-tuned, ensuring broad applicability. 2. Similarity Calculation: The cosine similarity (or Euclidean distance) between the embeddings of consecutive sentences is computed. 3. Rejection Sampling: If the similarity falls within a predefined interval [a, b], the sentence is accepted as "valid." Otherwise, the LLM is prompted to generate a new sentence, and the process repeats until a valid sentence is found or a maximum number of iterations is reached. This rejection sampling embeds a statistical pattern. 4. Optional PCA: Principal Component Analysis (PCA) can be optionally applied to reduce embedding dimensionality, potentially improving robustness for certain similarity measures.

Watermarked Text Detection: 1. Embedding & Similarity: The input text is segmented into sentences, and their embeddings are computed using the same model and PCA settings as during generation. Consecutive sentence similarities are calculated. 2. Soft Counting: Instead of a hard count of valid sentences (those within [a, b]), SimMark uses a "soft count." Sentences with similarities near the interval boundaries are partially counted, with a decay factor K controlling the smoothness. This enhances robustness against paraphrasing, which might shift similarities slightly outside the interval. 3. Soft-z-test: A statistical test (soft-z-test) is performed on the soft counts to determine if the text is likely LLM-generated. This test compares the observed soft count to an expected proportion (po) derived from human-written text.

Datasets

The paper evaluates SimMark using three standard English datasets:

  • RealNews: A subset of the C4 dataset.
  • BookSum: A collection of datasets for long-form narrative summarization.
  • Reddit-TIFU: A dataset of user-submitted stories.

The authors also mention using the Gemma3-4B model for additional experiments to demonstrate model-agnosticism.

Key Insights and Ideas

  • Sentence-Level is Key for Robustness: By operating at the sentence level and using semantic similarity, SimMark achieves better robustness against paraphrasing compared to token-level methods, which can be easily disrupted by rephrasing.
  • Black-Box Compatibility is Practical: The ability to watermark without access to LLM internals (logits) makes SimMark compatible with a wide range of models, including API-only services, which is a significant practical advantage.
  • Soft Counting for Paraphrase Resilience: The "soft counting" mechanism is a clever way to handle slight shifts in sentence similarity caused by paraphrasing. It allows for partial credit to sentences near the valid interval, preventing them from being entirely discarded and thus preserving the watermark signal.
  • PCA's Dual Role: PCA can be beneficial for Euclidean distance-based similarity by reducing noise, but it can be detrimental to cosine similarity, highlighting the importance of adapting preprocessing steps to the chosen similarity metric.
  • Generality Over Specialization: SimMark prioritizes general applicability by using off-the-shelf embedding models without fine-tuning, even if this means slightly lower performance in some highly specific scenarios compared to domain-specific methods.

Key Results

  • State-of-the-Art Robustness: SimMark consistently outperforms or matches other sentence-level watermarking methods (SemStamp, k-SemStamp) in robustness against various paraphrasing attacks, including bigram paraphrasing.
  • High Detection Performance: The paper reports high ROC-AUC and TP@FP scores across multiple datasets and paraphrasers, often achieving the best average performance. For instance, under Pegasus paraphrasing, SimMark achieved ROC-AUC scores around 97-99%.
  • Minimal Text Quality Degradation: SimMark shows comparable text quality metrics (perplexity, tri-gram entropy, semantic entropy) to unwatermarked text, indicating that the watermarking process does not significantly harm fluency or semantic diversity.
  • Efficiency: SimMark is 2-3 times more efficient in terms of sampling requirements compared to SemStamp and k-SemStamp. While it introduces a slowdown (around 5.2x for max_trials=25), this is considered practical, especially with optimized decoding backends.
  • Imperceptibility: Human evaluation suggests that SimMark's watermark is largely imperceptible, with participants being unable to reliably distinguish watermarked from unwatermarked text.

Limitations and Future Work

  • Rejection Sampling Overhead: While more efficient than prior sentence-level methods, SimMark's rejection sampling still introduces a latency overhead. Future work could explore batch sampling or parallel sampling to mitigate this.
  • Sophisticated Attacks: The paper acknowledges that SimMark might not be immune to more advanced adversarial transformations, such as interleaving watermarked text within larger unwatermarked content or attempts at reverse-engineering the algorithm.
  • Interval Dependency: The performance of SimMark depends on the choice of the predefined interval [a, b]. While consistent intervals worked well across datasets, adaptive strategies for setting these intervals dynamically could further enhance robustness and make reverse-engineering more difficult.
  • Token-Level vs. Sentence-Level Trade-offs: For very short generated texts, token-level methods might offer higher detection performance due to their finer granularity. SimMark's performance improves with longer generations.
  • Further Human Evaluation: A larger-scale human evaluation is suggested to confirm the imperceptibility of the watermark.

Summary updated

BibTeX
@inproceedings{dabiriaghdam2025simmark,
  author = {Amirhossein Dabiriaghdam and Lele Wang},
  title = {{SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure