AutoMark: Enabling Autoresearch to Discover Better LLM Watermarks

Thibaud Gloaguen, Robin Staab, Martin VechevarXiv 2026 · arXiv

TL;DR

AutoMark is the first framework for autonomous LLM watermarking research, leveraging frontier models to discover over 50 new watermarking schemes. By formalizing strict reliability criteria and building an automated verification harness, the authors discovered novel watermarks like ConcordMark and RotationFirst that significantly outperform existing state-of-the-art methods in robustness and text quality.

What Problem Does This Paper Address?

As LLM-generated content floods the internet, watermarking—embedding an invisible signal in text—has become a regulatory and commercial necessity. However, designing effective watermarks is a balancing act between three competing goals: detectability (how easily can we find the signal?), robustness (does the signal survive editing or paraphrasing?), and quality (does the watermark make the LLM's writing worse?).

Progress in this field has been limited by the slow pace of human researchers. Furthermore, existing evaluation methods are prone to "reward hacking," where a scheme might appear robust only because it has an unacceptably high false-positive rate (flagging human text as AI-generated). AutoMark addresses these gaps by automating the research loop while enforcing rigorous statistical safeguards.

Methods

The authors built an autoresearch harness that allows AI agents (like GPT-6 Astra, Opus 5, and Gemini-3.8 Flash) to iterate on watermarking code in an isolated sandbox.

Figure 1: Overview of our work. In an isolated sandbox, an agent iterates on a watermarking scheme. Submissions are first verified and, upon passing, evaluated and added to a leaderboard. We then manually decompose the discovered schemes into 5 components and ablate them for insights. Lastly, we evaluate two selected schemes on a held-out test set and find that they outperform prior methods.

The core innovation is a three-stage Automatic Verification pipeline:

  1. Distortion-freeness: Ensures the watermark doesn't change the LLM's output distribution in expectation (preserving the model's "natural" voice).
  2. Soundness over the key: Verifies that, on average, the false positive rate (FPR) is controlled.
  3. Soundness given a key: A novel, stricter requirement ensuring that any specific deployed key is unlikely to exhibit a high FPR on a fixed text distribution.

If a proposed scheme passes these tests, it is evaluated against a suite of attacks (paraphrasing, deletion, etc.) and ranked on a leaderboard.

Datasets

  • Verification Corpora: A mixture of the ELI5 dataset, WildChat (for LLM-generated text), and Wikipedia articles across 10 languages (including Russian, Arabic, Hindi, and Swahili).
  • Evaluation Models: Schemes were tested using LLAMA-3.1-8B and Qwen3.8-27B.
  • Attack Models: Paraphrasing and back-translation attacks were performed using GPT-5.4-mini and DeepSeek-V4 Flash.

Key Insights and Ideas

  • Beyond Hyperparameter Tuning: The agents didn't just tweak existing numbers; they discovered fundamentally new ideas. For example, they explored "Private Geometries," where watermark scores are aligned with random per-request directions in high-dimensional space.
  • The Power of Decomposition: The authors distilled the 50+ discovered schemes into 39 new atomic components across five categories: Unit, Seeding, Score, Logits Transform, and Detection.
Figure 3: Atomic components extracted from the agents’ watermarking schemes. Colors indicate the family ( Fresh race , Private geometry , Skeleton ) where components of the same family are interchangeable, and white components are compatible with every family. Skeleton schemes reuse the fresh-race scores and detectors (green edge). Components marked ∗come from prior work.
  • Gaussian Copula vs. Key Switching: Agents found that using a Gaussian copula to inject "true randomness" into the watermark score is strictly better than the "key switching" method used in prior work like TextSeal, offering a better trade-off between quality and robustness.
  • Residual Clocks: To handle repeated contexts (which usually break distortion-free guarantees), agents proposed "ResidualRace," which reuses previous "clocks" in a Gumbel race to maintain robustness without sacrificing detectability.

Key Results

The framework successfully pushed the Pareto frontier of LLM watermarking. Two schemes stood out:

  • ConcordMark: Achieved state-of-the-art robustness. It is "Pareto-superior" to all evaluated prior work, meaning it is better across almost all metrics simultaneously.
  • RotationFirst: Focused on quality, maintaining text diversity and perplexity nearly identical to unwatermarked models while still providing strong detection capabilities.
Figure 2: Submission history for each agent along two evaluated axes: robustness to paraphrasing and quality. Each dot corresponds to a submission and the solid lines are the best value so far. The baselines are in italic, and a parenthesis means the baseline fails our verification.

As shown in Figure 2, all tested agents managed to improve their submissions over time, eventually outperforming established baselines like AAR and SynthID, which actually failed the authors' stricter reliability checks.

Limitations and Future Work

  • Security: The paper focuses on reliability and robustness but does not deeply explore "security" against active adversaries trying to steal or spoof the watermark key.
  • Human Analysis: While the discovery was autonomous, the authors note that significant manual effort was still required to decompose and understand why the discovered schemes worked so well.
  • Scope: The current work is limited to distortion-free schemes and does not yet extend to multibit watermarking (embedding specific metadata) or "radioactive" data detection.

The authors suggest that future research should focus less on inventing individual schemes and more on building even more robust automated verifiers for these missing properties.

Summary updated

Abstract

With LLM watermarking being deployed commercially and now required by regulations, improving its reliability and effectiveness has become crucial. Yet, recent progress in the field of LLM watermarking has increasingly been driven by improving details of existing methods, an effort fundamentally limited by the pace of human researchers. In this work, we enable for the first time the autonomous discovery of new distortion-free state-of-the-art watermarking schemes. To enable this, we (i) establish strict criteria to ensure that watermarks are reliable (e.g., they do not have an unexpectedly high false positive rate), (ii) propose rigorous statistical tests to automatically evaluate whether a watermarking scheme satisfies our criteria, and (iii) design an evaluation suite to rank watermarks along three key dimensions: detectability, quality, and robustness. By running our framework with 3 frontier models (GPT-6 Astra, Opus 5, Gemini-3.8 Flash), we discover over 50 different watermarking schemes, including several that outperform prior works along all key dimensions. We complement this by a manual study of the discovered schemes, distilling the key ideas into smaller components, and individually studying the impact of each component across dimensions (detectability, quality, robustness) to better understand how the proposed schemes operate. Importantly, we find that the agents, on top of improving existing ideas, also discover fundamentally new ideas (e.g., aligning watermark scores with random per-request direction). Overall, our work establishes the first steps of fully autonomous watermarking research, enabling the discovery of more reliable and effective watermarks. Our code is available at https://github.com/eth-sri/automark, and a blogpost to visualize our results at https://www.sri.inf.ethz.ch/blog/automark.
BibTeX
@misc{gloaguen2026automark,
  author = {Thibaud Gloaguen and Robin Staab and Martin Vechev},
  title = {{AutoMark: Enabling Autoresearch to Discover Better LLM Watermarks}},
  year = {2026},
  eprint = {2609.37310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.37310},
}
Figure