A watermark hides a statistical signature in machine-generated text so the text can later be traced to its source. The best current watermarks are "distribution-preserving": averaged over secret keys, the watermarked model samples text with exactly the same probabilities as the original model. Multi-hop question answering is a hard setting for them. Here a model answers a question like "where was the director of Inception born?" by writing a step-by-step reasoning chain and using each step as a search query to fetch the next document. If a watermark changes even one entity in one reasoning step, the next search returns the wrong document, and every later step builds on the error.
We propose that in these systems, the right place to hide a watermark is set by what the retriever can and cannot see, not by how uncertain the model is. Our method samples with a watermark only at positions where every plausible next token would leave the retrieval query and the answer unchanged, such as connectives, function words and interchangeable phrasing. Everywhere else it decodes greedily, as the retrieval pipeline normally would. The scheme needs no training, works on frozen models, and can be verified from text alone. We also test a claim that has not been separated out before: the "quality tax" of distribution-preserving watermarks on reasoning may be mostly the cost of being forced to sample instead of decoding greedily. Our design is built to remove exactly that cost.
Core Claim
The existing literature disagrees with itself. Unbiased watermarks such as the reweighting schemes of Unbiased Watermark for Large Language Models, DiPmark and the tournament sampler SynthID-Text prove that no distortion occurs in expectation, and report unchanged perplexity, ROUGE and BLEU. Yet New Evaluation Metrics Capture Quality Degradation due to LLM Watermarking finds that GPT judges prefer unwatermarked text 67.5% of the time even for a distortion-free scheme. Watermarking Low-entropy Generation for Large Language Models shows that unbiased does not mean low-risk, because single samples vary a lot.
Our first claim is that much of this gap comes from what the unwatermarked baseline is. Iterative retrieval pipelines such as IRCoT decode greedily. A distribution-preserving watermark can only match the model's sampling distribution, never its greedy output. So any sampling variance at a token that feeds the retriever gets amplified by the retrieval loop.
Our second claim concerns the obvious fix, which is to watermark only where the model's next-token distribution is spread out (high entropy), as SynthID-Text and STA-M effectively do. We expect this fix to fail in retrieval-augmented generation. DRAGIN uses high token entropy as its signal that the model lacks information and needs to retrieve. High entropy on an entity token therefore marks the moment when the model is guessing a bridge entity, and that is exactly where randomizing the choice is most harmful.
We therefore gate the watermark with the retriever itself. A position is eligible only if the high-probability candidate tokens map to the same retrieval representation (for a sparse retriever such as BM25: all stopwords, or the same stemmed term) and the position lies outside the answer span. This gate is the proposal's single new ingredient, and every ablation below isolates it.
Relation to Prior Work
Several papers motivate the problem. IRCoT turns each generated reasoning sentence into a search query and reports gains of up to 15 F1 points and 11–21 points of retrieval recall. That same design makes the query a direct function of the sampled tokens. Chain-of-Retrieval Augmented Generation trains models to produce sub-query chains and documents that intermediate retrieval errors make later steps diverge. MuSiQue was built so that questions cannot be answered with disconnected shortcuts, which makes it the benchmark where a corrupted hop should hurt most. None of these papers considers watermarking.
The closest work is Echoes within the Reasoning (BiCoT). BiCoT observes that reasoning chains carry more entropy than final answers and embeds an ownership signature in the hidden states of reasoning tokens. It does this through bi-level fine-tuning and verifies using the top-40 log-probabilities. BiCoT is a model-ownership watermark: it modifies weights, is tested only on closed-book tasks, and cannot be checked from text. We share its intuition that the reasoning chain carries the signal. We differ in three ways: our method runs at decoding time on frozen models, it produces a per-output provenance signal detectable from text, and it is designed around retrieval.
Two papers supply methods we reuse. Inside eligible positions we run an existing unbiased sampler, DiPmark's reweighting or SynthID-Text's non-distortionary tournament, so the detection statistics come from those papers. STA-1 constrains the design: its evidence that resampling in low-entropy steps is risky is the reason we do not watermark at all outside eligible positions. Deductive Verification of Chain-of-Thought Reasoning and Faithful Chain-of-Thought Reasoning remind us that reasoning chains drift. We measure drift directly rather than assuming the watermark causes it.
Research Program
The work has three parts.
The first part is a decomposition study. We run IRCoT, and an adaptive-retrieval variant in the style of DRAGIN, under five decoding conditions:
- greedy decoding;
- plain sampling at the watermark's temperature;
- an unbiased watermark over all tokens;
- a biased green-list watermark over all tokens;
- the proposed gated scheme.
For every question we log the per-hop query, the retrieved set and the answer. This separates three costs: the sampling tax (greedy versus plain sampling), the watermark tax (plain sampling versus unbiased watermark), and the bias tax (unbiased versus biased watermark). It also lets us measure the mediator we hypothesize: the Jaccard overlap between each hop's retrieved documents and the greedy run's documents for the same hop.
The second part builds the gate. At each step we take the smallest candidate set holding probability mass $\tau$ and apply the retriever's own tokenizer, stopword list and stemmer to it. The position is eligible if every candidate maps to the same retriever-visible term, or to no term. At eligible positions we sample with the watermark. At ineligible positions we take the greedy token.
The answer span is marked by the pipeline's own answer delimiter ("So the answer is:"), and is never watermarked. For dense retrievers the analogous test compares query embeddings under each candidate. We treat that version as secondary, because it costs one embedding call per candidate.
Detection counts green-token or g-value statistics only at positions a verifier can recompute as eligible from the text. That requires the verifier to reconstruct eligibility without the model. We therefore also test a model-free approximation of the gate that uses only the text's own stopwords and stems.
The third part characterizes detection. We plot true-positive rate at 1% false-positive rate against chain length and number of hops, because the gate shrinks the number of usable tokens. We then measure robustness under the 20–30% edit rates that DiPmark reports surviving on open text.
Baselines
The naive reference points come first. Greedy IRCoT with no watermark is the accuracy ceiling we aim to match. Plain sampling IRCoT is the fair reference for any distribution-preserving scheme, and on its own it answers whether "watermark degradation" is really sampling degradation. For the IRCoT family, IRCoT reports gains of up to 15 F1 over one-step retrieval, and on 2WikiMultihopQA DRAGIN reports F1 of 0.3931 with LLaMA2-13B against 0.3076 for FLARE. We rerun these systems rather than copying their numbers, because we need exact retriever and prompt parity. No reported result was found on Lacuna for any watermark applied inside a multi-hop retrieval loop.
Off-the-shelf DiPmark or SynthID-Text over the whole IRCoT output is the strongest existing competitor. These schemes are proven distribution-preserving, cheap (SynthID reports about 0.6% latency overhead), and are what a practitioner would deploy today. STA-M-style entropy gating, which watermarks only high-entropy steps, is the strongest existing low-risk variant and the direct rival to our gate. A biased green-list watermark marks the high-degradation end.
Two ablations isolate the key ingredient:
- Answer-span-only exclusion removes the retriever-invariance test and keeps only the answer mask.
- Named-entity exclusion skips tokens a tagger labels as entities, a cheaper heuristic stand-in for "don't touch bridge entities."
All systems share the same model (LLaMA2-13B, plus Flan-T5-XL as in IRCoT), the same BM25 index, prompts and few-shot examples, and matched temperature for every sampling condition. Each gets an equal tuning budget over its single strength or threshold parameter on a held-out development split. Each is reported over at least five watermark keys.
The gated scheme beats the field if two things hold on MuSiQue and 2WikiMultihopQA. Its answer F1 and supporting-paragraph recall must be statistically indistinguishable from greedy IRCoT. And at matched true-positive rate at 1% false-positive rate, it must be significantly above both full-output DiPmark/SynthID and entropy gating.
Why not just…?
Why not just turn on SynthID-Text or DiPmark inside IRCoT? These schemes guarantee equality with the sampling distribution, while IRCoT's reported gains come from greedy decoding. The low-entropy study shows that unbiased single samples can still be wrong (STA-1), and in a retrieval loop one wrong query token changes the documents fetched. This is a real threat only if plain sampling costs little on these benchmarks. The decomposition study settles it directly. If sampling and watermarked sampling both match greedy, our method is unnecessary and we will report that.
Why not just gate on entropy, as STA-M and SynthID already do? The competing hypothesis is DRAGIN's: in retrieval-augmented generation, high entropy on content tokens signals missing knowledge. We test this by measuring what fraction of watermarked positions under entropy gating fall on bridge-entity or query-content terms, and whether those positions predict retrieval divergence from the greedy run. If entropy gating rarely touches such terms and matches greedy accuracy, our gate adds nothing beyond it and we would drop the claim.
Why not just watermark only the final answer? Multi-hop answers are short entities with almost no entropy. Detection tests need many tokens, and DiPmark notes this difficulty for short snippets. The answer-only condition is included and should show near-chance detection.
Why not use BiCoT? BiCoT requires fine-tuning with bi-level optimization, verification with top-40 log-probabilities, and closed-book settings. It cannot mark outputs of a frozen model inside a retrieval loop, and it cannot be checked from text. It answers a different question, model ownership, and is not run as a baseline.
Won't a stopword-heavy watermark be easy to paraphrase away? Possibly. This is the proposal's main risk. We report robustness under token substitution and under full-sentence paraphrase, and we compare against full-output DiPmark at equal edit rates.
Decisive Experiments
We evaluate on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC with the IRCoT retrieval setup. Metrics are answer exact match and F1, supporting-paragraph recall, per-hop retrieved-set overlap with the greedy run, detection true-positive rate at 1% false-positive rate with AUC, and pairwise LLM-judge coherence preference following New Evaluation Metrics.
Four results decide the proposal:
- Is the tax mostly sampling? If the accuracy drop between greedy and plain sampling accounts for most of the drop under the unbiased watermark, the literature's contradiction is resolved in favor of our explanation.
- Does retrieval divergence carry the harm? If per-hop overlap with the greedy run statistically explains the F1 loss, and the effect grows with hop count on MuSiQue's 2–4 hop splits, the retrieval-amplification mechanism is confirmed.
- Does entropy gating watermark the dangerous tokens? If it places marks on query-content terms far more often than our gate does, and does worse on MuSiQue, retriever-aware gating is shown to be the necessary ingredient.
- Is enough signal left? If gated chains of typical IRCoT length still reach high true-positive rate at 1% false-positive rate, the scheme is practical. If they do not, the honest conclusion is that multi-hop reasoning chains are too short to watermark safely, and that is itself worth reporting.
References
- Unbiased Watermark for Large Language Models
- DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models
- Scalable watermarking for identifying large language model outputs
- New Evaluation Metrics Capture Quality Degradation due to LLM Watermarking
- Watermarking Low-entropy Generation for Large Language Models: An Unbiased and Low-risk Method
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Chain-of-Retrieval Augmented Generation
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thought
- Deductive Verification of Chain-of-Thought Reasoning
- Faithful Chain-of-Thought Reasoning