Watermarking the Starting Noise of Speaker-Conditioned Speech Diffusion

Modern voice cloning systems can imitate an unseen speaker from a few seconds of audio, and detectors that look for artifacts are losing. On the ASVspoof 5 challenge, standard detectors reach error rates of 29–36%. The alternative is proactive: the generator hides a machine-readable signature (a watermark) in its own output. For speech, however, almost all such work targets token-based generators, meaning language models over compressed audio codes. Many strong cloning systems are instead diffusion models, which produce a spectrogram by repeatedly denoising random noise. For diffusion models there is an attractive option already shown to work for images: write the watermark into the random noise the generator starts from. Later, run the generator backwards on a suspect clip to recover that noise and read the bits.

This note argues that the speech version of this idea is not a straight port, and that speaker variability is the reason. In speaker-adaptive diffusion text-to-speech (TTS), sampling starts not from pure noise but from noise added to a prior mean. That mean is computed from the text and from a style vector describing the target voice. A detector holding only a waveform does not know the text, the speaker, or that mean. It must re-estimate all three from the audio before it can run the model backwards. The watermark can only be as reliable as that estimate, and the estimate gets worse for exactly the voices the model handles worst. We propose to build this detection-by-inversion pipeline for a multi-speaker diffusion TTS system and to place watermark bits in the parts of the noise that survive both the inversion error and speaker variation. The central test is whether the result survives a diffusion-based "scrubbing" attack better than an ordinary post-hoc audio watermark. If it does not, the idea should be dropped.

Core Claim

We take a speaker-conditioned diffusion model of the Grad-StyleSpeech type. There, $\mu$ is the prior mean built from phonemes, durations and speaker style, the sampler starts at $x_T = \mu + z$ with $z \sim \mathcal{N}(0, I)$, and a deterministic probability-flow sampler maps $x_T$ to a mel-spectrogram. A watermark written into $z$ is recovered as

$$\hat z = \Phi^{-1}_{\hat c}(\hat x_0) - \hat\mu .$$

Here $\hat x_0$ is the mel-spectrogram recomputed from the received waveform, $\hat c$ and $\hat\mu$ are the conditioning and prior re-estimated from that waveform (transcript from speech recognition, speaker style from the audio itself, durations from forced alignment), and $\Phi^{-1}_{\hat c}$ is the reverse ODE run under that estimated conditioning.

The recovery error has two parts. One is numerical and channel drift inside $\Phi^{-1}$. The other is conditioning error $\mu - \hat\mu$, which is new relative to image work and is speaker-dependent.

We make three claims. First, conditioning error, not ODE drift, dominates bit errors in multi-speaker zero-shot cloning. Second, placing bits only in the high-magnitude tails of $z$ (the T2SMark idea) and only in time–frequency coordinates whose recovery is stable across speakers gives bit accuracy close to the oracle-conditioning case. Third, because such a watermark shapes the whole generated sample rather than adding a high-frequency perturbation, it survives regeneration by a speech diffusion model that strips post-hoc watermarks.

The single-speaker setting plays a specific role here: it is the easy case where the detector knows the conditioning. We expect robustness measured on single-speaker corpora to overstate what happens across many speakers.

What Prior Work Does and Leaves Open

The mechanism comes from image diffusion. T2SMark embeds bits in the tails of the initial Gaussian noise and leaves a random "undecided zone" near zero. It reports bit accuracy 0.9754 under attack, against 0.5024 for PRCW, while keeping generation diversity (LPIPS 0.7069 vs 0.7072 unwatermarked). Guidance Watermarking instead steers intermediate denoising steps with gradients from a watermark detector. It reports that this spreads watermark energy across the spectrum, whereas decoder-based watermarks concentrate it in fragile high frequencies. Both work unconditionally or with known text prompts on images. Neither faces a detector that must guess a speaker-dependent prior, and neither passes through a vocoder.

The generator comes from Grad-StyleSpeech. It starts its reverse process from a style-conditioned prior and reports MOS 4.18, similarity MOS 3.83 and speaker-embedding cosine similarity (SECS) 87.51% on LibriTTS. It has no traceability. Its strong prior matters to us: because $\mu$ already fixes coarse spectral structure, the noise $z$ may control relatively little of the output. This limits how much watermark can live there, and we measure it directly.

Three speech watermarking systems motivate the gap. TraceableSpeech watermarks a discrete codec inside a VALL-E-style pipeline. SSR-Speech adds a frame-level watermark codec to autoregressive editing. A Watermark for Auto-Regressive Speech Generation Models biases token sampling. The last of these also documents the failure we expect to see in continuous form: detection drops sharply once audio is vocoded and re-tokenized, and recovering it costs fidelity (Fréchet Audio Distance rising from 0.0051 to 0.0494 at a bias of 2.0). All three rely on discrete tokens.

Timbre Watermarking protects reference recordings with an STFT-domain watermark that survives GAN-vocoder cloning pipelines. Two papers constrain the design. DiffWA shows that a diffusion model trained on clean data removes image watermarks (bit error rate above 0.4) while keeping PSNR above 31 dB. That is the attack any speech watermark must now face, though DiffWA itself was never tested on speech. DLD-AE shows that latent diffusion can separate speaker from content in pretrained speech features. We draw on that when re-estimating speaker style from possibly degraded audio.

Research Program

We first train a Grad-StyleSpeech-style model on LibriTTS with a deterministic ODE sampler and a fixed HiFi-GAN vocoder. We also train a single-speaker twin on LJSpeech. Before any watermarking, we measure how much of the output the noise actually controls. We invert clean generations under oracle conditioning and record, per time–frequency coordinate, the correlation between original and recovered $z$. Repeating this with conditioning estimated from the waveform isolates the conditioning error. If recovery under oracle conditioning is already poor after vocoding, the approach has too little capacity and we stop.

We then build the blind detector. It transcribes the audio, force-aligns the transcript to get durations, and extracts a style vector with the model's own mel-style encoder. As a variant, the style vector is first cleaned with a DLD-AE-style speaker encoder. The detector then inverts and thresholds $\hat z$ at the tail boundaries. Keys follow T2SMark's two-level scheme, so each utterance's noise remains pseudo-random.

Bit placement is the one new design ingredient. We keep only coordinates whose recovery reliability, measured over a held-out pool of training speakers, stays above a threshold for most speakers rather than on average. We repeat bits across frames so the watermark tolerates cropping, as Timbre Watermarking does. Placement is chosen once, on training speakers, and frozen before any test speaker is seen.

Baselines

The naive reference point is the unwatermarked Grad-StyleSpeech-style model evaluated by the same detector. It fixes the false-positive floor and the fidelity ceiling. We will check our reproduction against the reported LibriTTS figures of MOS 4.18, similarity MOS 3.83 and SECS 87.51% from Grad-StyleSpeech.

The strongest practical competitor is a post-hoc waveform watermark applied to the same model's output. We use the WavMark-style post-hoc scheme that TraceableSpeech used as its baseline, together with the robust STFT embedder of Timbre Watermarking. It is the strongest competitor because it is speaker-agnostic, needs no inversion, and its authors report 100% extraction through cloning pipelines and MP3. For fidelity, TraceableSpeech reports PESQ 3.19 for the post-hoc baseline against 3.64 for its own in-codec watermark, and about 76% extraction for the post-hoc baseline under resplicing. For fairness, we give it the same payload in bits per second, retrain its embedder on the same LibriTTS training split, and give it the same augmentation set and tuning budget.

We also port Guidance Watermarking to our model. It uses the same detector architecture and the same number of sampling steps, and we report its wall-clock generation cost.

As a passive reference, we run the flow-matching DiT residual detector of Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals. It reports 6.54% equal error rate (EER) on ASVspoof 5 Eval and 13.84% on In-The-Wild. It answers a different question and cannot identify which system produced a clip, so it bounds rather than competes.

Two ablations isolate our ingredient. The first is uniform tail placement, which is T2SMark ported unchanged. The second is our placement with oracle conditioning, which gives an upper bound.

Beating the post-hoc baseline means matching its bit accuracy on undistorted and compressed audio and exceeding it by a clear margin after speech-diffusion scrubbing, at no worse MOS or SECS. Beating the ported T2SMark means higher worst-decile per-speaker bit accuracy.

Why not just…?

Why not just add a post-hoc audio watermark to the diffusion output? This is the real threat. Under compression and cropping alone we expect it to perform as well as our method or better, because it avoids inversion error entirely. Its expected weakness is regeneration. DiffWA's result that clean-data diffusion strips watermarks at little perceptual cost was shown only on images; for speech there is no reported result found on Lacuna. Our main experiment builds that speech attack. If post-hoc watermarks survive it about as well as ours, the in-generation approach has no reason to exist, and we will report that outcome.

Why not just steer generation with detector gradients (guidance watermarking)? It avoids the need to guess the conditioning, since the detector reads the output directly, and its energy-spreading property may give similar scrubbing robustness. Its costs are a backward pass per sampling step and a learned detector that inherits speaker bias from its training data. We run it head-to-head and compare robustness per unit of generation time. We also compare per-speaker spread, because a learned detector could perform worse on under-represented voices.

Why not just train a better passive detector? Passive detection works against uncooperative generators, which watermarking cannot address. But ASVspoof 5 baselines sit at 29–36% EER (ASVspoof 5), and even the DiT-residual system reaches 13.84% EER in the wild while needing 2.37 seconds per utterance. Those error rates are far from what attributing a clip to a specific operator requires.

Why not just fine-tune the vocoder to emit the watermark? This is the continuous analogue of TraceableSpeech's codec watermark. It is simple, and we include it as a secondary baseline. The Guidance Watermarking paper reports that watermarks inside a decoder concentrate energy in high frequencies. Those are exactly the frequencies that compression and diffusion purification remove, so we expect it to behave like the post-hoc baseline under attack.

Proposed Experiments and Decisive Checks

We generate zero-shot clones for unseen LibriTTS test speakers and for VCTK's accented speakers. Each clip passes through the transmission channels ASVspoof 5 uses (MP3, neural codecs, telephone-band simulation), plus cropping to 10% and resplicing. Each clip also goes through the speech scrubbing attack: a DiffWA-style conditional diffusion model trained on bona fide LibriTTS, which partially noises and re-denoises the audio under a waveform-distance constraint. We sweep the attack strength and keep only levels where scrubbed audio stays within 0.1 MOS and 2 SECS points of the input, so the attack is not simply destroying speech.

We report bit accuracy, true-positive rate at fixed false-positive rates of 1% and $10^{-4}$, MOS, SECS and PESQ. Every result is broken down per speaker and by gender, given LibriTTS's documented imbalance (LibriTTS).

Four checks decide the project. The first is the scrubbing curve: our watermark must keep usable bit accuracy at attack strengths where the post-hoc watermark falls toward chance. The second is the error decomposition: the gap between blind and oracle conditioning should dominate the gap from vocoder drift, and prior-aware placement should close most of it. If it does not, the speaker-variability claim is wrong. The third is the single-to-multi-speaker comparison: the same scheme should look nearly perfect on the LJSpeech twin and lose accuracy on the worst decile of LibriTTS and VCTK speakers. This quantifies how much single-speaker evaluations overstate robustness. The fourth is the capacity measurement from the first stage, which tells us how many bits per second the noise can carry before fidelity, measured by MOS and diversity across seeds, starts to fall.