Phase-Free Detection of Audio Token Watermarks via Domain-Coherent Clusters

Speech generators that produce discrete audio tokens can be watermarked by nudging which tokens they pick. A detector later re-encodes the audio into tokens and checks for the nudge. This works until someone crops a few milliseconds off the front or slightly changes playback speed. The audio codec, the neural compressor that turns waveforms into tokens, chops the signal into fixed frames, typically tens of milliseconds long. A crop shifts where those frame boundaries fall, every re-encoded token changes, and the watermark disappears. Today the detector repairs this by trying many candidate offsets. That costs one codec pass per offset, and every extra try is another chance for a false alarm.

We propose making the watermark immune to frame position by construction. The idea is borrowed from how liquid-crystalline polymers order themselves. Cooling through an intermediate phase builds ordered domains that are much larger than the molecular spacing, and those domains, not individual molecules, carry the structure into the solid. In the same way, we bias generation so that the token's cluster label stays constant over short domains of several frames. The secret key then lives in the sequence of labels from one domain to the next, not in which token sits at which frame index. Any re-encoding grid, at any sub-frame offset or modest speed change, still lands most of its frames inside a domain and reads the right label. Detection becomes a single codec pass followed by a test on runs of identical labels, with no offset search.

Core Claim

Two recent methods handle substitution errors in re-encoding by watermarking clusters of acoustically similar tokens rather than individual tokens. Robust Distortion-Free Watermark for Autoregressive Audio Generation Models builds the clusters by k-means (grouping points around a fixed set of centers) on codec embeddings and samples so the overall output distribution is unchanged. Hidden in Plain Tokens builds them by community detection on a graph that records which tokens get confused with which across repeated codec passes.

Both methods treat temporal registration as unsolved. The first falls back on searching over multiple offsets. The second names cropping and speed changes as its main limitation. Both clusterings are static: they absorb a token being swapped for its neighbor, but they cannot absorb the frame grid moving. Our claim is that the missing ingredient is a length scale. If cluster labels are coherent over domains of $\ell$ frames, with $\ell$ larger than the codec's receptive field measured in frames, a shifted re-encoding preserves most labels. Labels sit inside domains, and only frames straddling a boundary are corrupted.

A detector that segments the re-encoded label stream into runs and tests keyed transitions between consecutive runs needs no knowledge of frame phase. It tolerates speed changes because runs stretch rather than break. The price is lower entropy within a domain, and that trade-off is exactly what this work must measure.

Concretely, let the re-encoded first-codebook tokens be mapped to cluster labels. We merge runs shorter than $r_{\min}$ frames into their neighbors, and call the resulting run labels $a_1,\dots,a_M$. Let $G(a;\kappa)$ be the green set of clusters that the secret key $\kappa$ assigns as successors to label $a$, covering a fraction $\gamma$ of all clusters. The detector computes

$$z=\frac{\sum_{m=2}^{M}\mathbf{1}[a_m\in G(a_{m-1};\kappa)]-\gamma(M-1)}{\sqrt{\gamma(1-\gamma)(M-1)}},$$

which is approximately standard normal for unwatermarked audio. The test uses only the order of runs, never their absolute positions.

The fraction of frames whose label survives a shift is the main falsifiable quantity. If $R$ is the number of frames on which a codec frame's encoding depends, a simple counting argument predicts

$$\rho(\ell)\approx \max\!\left(0,\ \frac{\ell-R}{\ell}\right),$$

independent of the sub-frame offset. The reasoning is that, of the $\ell$ frames in a domain, only those whose receptive field crosses a domain boundary are at risk.

Relation to Prior Work

The materials literature serves as a design lens, not as a method to simulate.

Non-covalent planarizing interactions yield highly ordered and thermotropic liquid crystalline conjugated polymers shows that cooling a polymer through nematic and then smectic phases heals defects. Conductivity rises from 18 to 55 S/cm and device yield reaches 100%, with a narrow spread of performance. The lesson we take is that order carried by domains makes outcomes insensitive to local misregistration. We also borrow the hope of a narrow spread, here meaning detection strength across speakers.

Non-Covalent Interactions and Helical Packing in Thiophene-Phenylene Copolymers acts as a constraint. Small structural changes divert ordering into unintended phases. Our analog of an off-target phase is degenerate, repetitive audio when persistence is pushed too hard, so we treat persistence strength as a dial to sweep, not a fixed choice.

Effects of composition and crystallographic ordering on the ferromagnetic transition in Ni Co Mn In magnetic shape memory alloys finds that annealing temperature, not annealing time, sets the degree of order. This suggests a concrete hypothesis. The strength of the persistence bias, not the nominal domain length $\ell$, may be what controls robustness, and our sweep separates the two.

On the signal side, the two cluster watermarks above supply both the clustering method and the baselines we must beat. SIGMark solves the video version of the problem differently. It recovers the grouping phase of the video autoencoder with optical flow and a sliding-window match against global pseudo-random codes. That is a search, made cheap, rather than an invariance. XAttnMark spreads message bits across time in a post-hoc embedder, meaning one applied to finished audio rather than during generation, but it still loses attribution accuracy under tempo changes. AudioMarkBench documents that neural codecs strip post-hoc marks and that robustness differs by speaker sex and language. That benchmark gives us both a threat model and a fairness test.

Research Program

We work with the speech language models and codecs used by the baselines: SpiritLM and SpeechGPT as generators, and EnCodec and Mimi as tokenizers. The watermark acts only on the first, most semantic codebook layer of each codec.

The first step is to measure the receptive field $R$ empirically for each codec. We shift waveforms by every sub-frame offset and record which neighboring frames' tokens change.

The second step builds the clusters. We use both k-means and community detection on a confusion graph. We add one change: the confusion graph is computed from shifted re-encodings rather than repeated passes at the same alignment, so clusters absorb the substitutions that offsets actually cause.

The generator keeps a current target cluster. At each step it adds a logit bonus $\beta$ to tokens in that cluster, where a logit is the model's pre-softmax score for a token. After the run has lasted $\ell$ frames, the generator draws the next target from $G(\text{current};\kappa)$. Both $\beta$ and $\ell$ are swept. Optionally, $\beta$ follows a schedule that starts soft and tightens; this is our "cooling" variant, which we include only if it improves the quality-robustness trade-off.

The detector re-encodes the audio once, merges short runs, and computes $z$.

We will also check whether the $\rho(\ell)$ prediction holds. Large deviations would indicate that codec context, rather than local receptive field, drives the token changes.

Baselines

The naive reference point is the per-token cluster watermark of Wu et al. (ALIGNED-IS) or Milis et al., detected at a single assumed offset of zero. We expect it to collapse under sub-frame shifts, and that collapse shows the size of the gap we are trying to close.

The strongest competitor is the same cluster watermark with multi-offset detection, the fallback Wu et al. describe. It tries $H$ candidate offsets and takes the best score. Fairness requires two corrections. First, the false-positive rate must be calibrated for the maximum over $H$ tests using empirical nulls on unwatermarked LibriSpeech audio, so that "1% FPR" means the same thing for both methods. Second, both methods are reported at matched detector compute, measured in codec forward passes.

The reported numbers for this baseline are TPR above 0.90 at 1% FPR across MMW, Dolly CW, Longform QA and LibriSpeech, plus survival at 0.9× and 1.1× speed (Robust Distortion-Free Watermark for Autoregressive Audio Generation Models). For the community-detection variant, the reported figure is a detection strength $-\log p = 42.47$ against 8.51 for a token-level baseline (Hidden in Plain Tokens). We found no reported result on Lacuna for either method under arbitrary sub-frame offsets or front crops with single-pass detection, so we will measure that baseline ourselves.

Beating it means matching its calibrated TPR at 1% FPR using one codec pass instead of $H$, or exceeding it at equal compute under crops and speed factors of 0.8× to 1.2×. Both must hold with no more than an agreed loss in quality: word error rate under a fixed speech recognizer, and the JND-trained metric of A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences.

A post-hoc reference is XAttnMark, which reports 99.19% detection and 93% attribution across 16 transforms on MusicCaps. We run it on the identical attacked audio to place in-generation and post-hoc marks on one axis.

The key ablation sets persistence to zero ($\beta=0$) while keeping run-based detection and transition keys. If that variant also survives shifts, then domains are not doing the work and the claim fails.

Why not just…?

Why not just search over offsets? This is the real threat, and the proposal stands or falls on beating it. Search is linear in offset resolution, so its cost grows as offsets get finer. It also raises false positives, because each extra window is another chance for a pseudo-random match. That problem is documented for frame-based schemes by Wang et al.. Speed changes add a second dimension to the search. Our compute-matched, calibration-corrected comparison settles the question directly. If search at modest $H$ matches our method with no quality cost, we will report that domain coherence is unnecessary for audio and limit the claim to cases with speed perturbation.

Why not just add a synchronization pilot or repeat a post-hoc mark? Repetition and pilots work for additive marks, but AudioMarkBench shows that neural codecs such as EnCodec strip post-hoc marks and that additive schemes like WavMark fail under MP3. In-generation token marks survive codecs precisely because they live in the tokens. A pilot embedded in tokens would face the same frame-shift problem we are addressing. The XAttnMark comparison on the same attacks makes this trade-off visible rather than assumed.

Why not just make the codec shift-equivariant, or fine-tune it? Doing so changes the deployed model and its audio quality. It also gives up the gradient-free, plug-in property that motivates Hidden in Plain Tokens. Strided convolutional encoders are not shift-equivariant below their stride, so fine-tuning cannot remove the frame grid. Our method leaves every model frozen.

Why not just seed the key on the previous token's cluster? That is the ablation above. Under a shift, the previous frame's label is exactly what gets corrupted, unless labels persist across frames. The $\beta=0$ arm tests this.

Proposed Experiments

We generate speech on LibriSpeech, LibriTTS, MMW, Dolly CW and Longform QA prompts, and music on MusicCaps where the codec supports it. Attacks are a sweep over every sub-frame offset, front and random crops to 2 to 10 seconds, speed changes from 0.8× to 1.2× with and without pitch preservation, MP3, and codec re-encoding. For each attack we report TPR at 1% FPR, $-\log p$, the number of codec passes, word error rate, and the perceptual metric, all as functions of $(\ell,\beta)$.

The decisive checks are four. First, the measured label survival should track $\rho(\ell)$ and stay flat across offsets. Second, single-pass detection at the worst-case half-frame offset should match calibrated search at one $H$-th of the compute. Third, the $\beta=0$ ablation should fail under shifts. Fourth, the quality cost at the operating point should stay within the agreed bound.

A further test uses AudioMarkData, a benchmark with balanced speaker sex, language and age groups. Stratifying detection strength by these groups tests whether domain coherence narrows the demographic gap that AudioMarkBench reports, the analog of the narrow device spread seen in liquid-crystal templating. We treat this as a secondary hypothesis and report it whether it holds or not.

We would abandon the approach if any of three things happens. The first is that calibrated multi-offset search matches it at equal compute. The second is that robust settings require persistence strong enough to raise word error rate noticeably. The third is that label survival depends on offset in a way the receptive-field model cannot explain.