Detectors that find the few edited words inside an otherwise genuine recording look nearly perfect on their home benchmark and collapse on the next one. A speech language model that tags edited words reaches 97.51% word-level F1 on one dataset and close to 0% on another (Can LLMs Help Localize Fake Words in Partially Fake Speech?). We argue that this is benchmark contamination in a form the field has not named. Every static forgery dataset is built by one recipe, and that recipe leaves label-correlated fingerprints the detector can memorize instead of the evidence we care about. Examples include antonym edits written by ChatGPT, one speech synthesizer, a fixed splicing method, and codec processing applied only to the fake part. Language-model researchers met the same problem and answered it with procedurally generated tests whose structure varies on every draw (DyVal). General game playing offers a ready template for generating such tests richly: a grammar of small rule units, called ludemes, that a single engine recombines into thousands of distinct games (Ludii).
We propose a ludeme-style grammar for partial audio forgeries. Each instance is assembled from independently sampled factors, so that every nuisance factor can be made statistically independent of the "fake" label. We pair the grammar with a regression measure, which we call shortcut share, that reports how much of a detector's output is explained by nuisance factors rather than by the edit itself. Together these allow three things. First, we can diagnose existing detectors. Second, we can train on decorrelated data. Third, we can ask a sharp question about the cross-modal gap: once lexical shortcuts are removed, does feeding a language model the transcript help localization at all, and does better alignment of audio features to the language model's input space change the answer?
Core Claim
Out-of-domain failure of partial-forgery localizers is mostly caused by shortcuts that the benchmark construction makes available, not by an inability to perceive splice boundaries. A generator that crosses nuisance factors independently will expose these shortcuts quantitatively. On such data, shortcut share measured on existing detectors should predict their known cross-dataset drops. Training on decorrelated compositions should transfer better to unseen real benchmarks than training on an equal amount of conventionally built data. The novelty does not lie in a new detector. It lies in treating forgery data as a compositional rule space with held-out combinations, the way general game playing holds out unseen games, and in importing contamination-style residualization into forensics.
Research Program
The grammar has eight forgery ludemes. They are the edit operation (replace, insert, delete, and a null edit that resynthesizes a word without changing its text), the lexical relation of the edit (antonym, synonym, neutral, unchanged), edit position within the utterance, edit duration, the synthesizer, the splice method, background processing, and the speaker. The synthesizers are VITS and YourTTS as used in AV-Deepfake1M, plus VoiceCraft, SSR-Speech and Audiobox as used in PartialEdit. The splice methods are hard cut, crossfade, model in-filling, and an authentic splice of the same speaker's real speech. That last option creates a seam with no synthesis. Background processing covers codec passes over the whole utterance (the PartialEdit-Codec condition), additive noise, and reverberation, which matter for seam artifacts per An Investigation of Spoofing Speech Detection Under Additive Noise and Reverberant Conditions.
Two ingredients break the shortcuts. The null edit and the authentic splice separate "synthesized" from "seamed" from "lexically changed." In addition, genuine utterances are deliberately sampled to contain negative-polarity words at the same rate as fakes do. This removes the polarity cue documented by the speech-LLM work. Sources are VCTK and other read-speech corpora, with forced alignment providing word boundaries.
Shortcut share is defined on a decorrelated evaluation draw, where nuisance factors $z$ (lexical relation, position, synthesizer, codec, and so on) are independent of the true label $y$. On that draw, we fit the detector's per-word logit $s$ by least squares on $z$ alone and on $(z, y)$:
$$\text{ShortcutShare} = \frac{R^2(s \mid z)}{R^2(s \mid z, y)}.$$
Here $R^2(s \mid \cdot)$ is the fraction of logit variance explained by the listed regressors. A detector that responds only to the edit scores near zero. One that responds to cues present in its training recipe scores high even though those cues carry no label information here. This measure borrows directly from Recall Residualisation, which used ordinary-least-squares residualization to show that 97% or more of apparent forecasting skill was recall in 19 of 20 cases. We share its linearity limitation. As a nonlinear check, we therefore also report a gradient-boosted version of the regression.
On the modeling side, we revisit the speech LLM of Can LLMs Help Localize Fake Words in Partially Fake Speech?, which uses a WavLM encoder, a single linear projector, and TinyLlama. We hypothesize that the lexical shortcut is partly a modality-gap effect. CALF shows that linear projections leave continuous tokens segregated from text tokens in feature space, so the language model may simply find the text easier to use than poorly aligned audio. We test a CALF-style variant that adds intermediate-layer feature regularization and output consistency between an audio-only branch and an audio-plus-text branch. The consistency term pushes the model's tagging decision to be recoverable from audio alone.
Baselines
The naive reference point has two parts. One is a text-only classifier, the same TinyLlama fine-tuned on transcripts alone to emit [fake] tags. The other is a position prior that predicts edits at the most frequent training location. Their scores on any benchmark give the ceiling of pure lexical and positional leakage, and a "real" detector must clearly exceed them on decorrelated draws.
The strongest language-based competitor is SLM-AT from Can LLMs Help Localize Fake Words in Partially Fake Speech?. It reports 97.51% WordF1 in-domain on AV-Deepfake1M and 90.79% in-domain on PartialEdit, and near 0% when trained on AV-Deepfake1M and tested on PartialEdit. This is the right competitor because it is the only existing system that uses the transcript, which is exactly the component our hypothesis targets.
The strongest acoustic competitors are three systems. The first is the coarse-to-fine proposal refiner CFPRF, which reports 7.41% EER on ASVS2019PS and 99.23% mAP on HAD. The second is the XLSR-SLS detector as evaluated in PartialEdit: 2.55% EER in-domain on PartialSpoof, 12.95% on PartialEdit, and 23.72% on PartialEdit-Codec. The third is the bilevel detector of PhonemeFake, which reports a 91% relative EER reduction on its own dataset; we found no absolute EER on Lacuna to quote. All systems are retrained under the same protocol: the same training audio hours, the same WavLM or XLSR front-end size where architectures allow, and the same hyperparameter search budget of 20 trials tuned on a validation draw of the training distribution.
Two ablations isolate the key ingredients. The first trains every model on grammar data generated with factors correlated as in AV-Deepfake1M and PartialEdit (antonym edits only, one synthesizer per split, codec only on fakes) versus the decorrelated grammar, at equal size. This isolates decorrelation from data quantity and diversity. The second compares the linear projector with the CALF-style aligned projector, and both with the transcript removed. This isolates alignment from simple transcript dropping.
Beating the baselines means two things. Grammar-trained models must cut shortcut share below the text-only ceiling. They must also improve zero-shot WordF1 and EER on held-out real benchmarks over the same architectures trained on equal-size static data. We set the bar as non-overlapping 95% bootstrap intervals.
Why not just…?
Why not just train on several static datasets at once? Mixing AV-Deepfake1M, PartialSpoof and PartialEdit adds generator diversity, and this is a real threat. Mixing does not, however, remove correlations that all these datasets share, such as LLM-written semantic edits and codec processing applied only to fake regions. The correlated-versus-decorrelated ablation, run at matched size and matched synthesizer set, settles the question directly. If mixing transfers as well as decorrelation, we will report that the grammar's value is diagnostic rather than as training data.
Why not just drop the transcript and use an acoustic localizer? Removing text removes the lexical shortcut by construction. CFPRF may well match any speech LLM on seam detection. We take this seriously enough that it reshapes the modeling question: we do not assume language helps. We measure, on decorrelated draws, whether aligned audio-plus-text beats audio-only. A null result here would itself be useful, because it would show that the language model contributes only shortcuts. Acoustic detectors are not immune to shortcuts either, as the codec result in PartialEdit shows, so the grammar remains necessary for them.
Why not just hold out one generator at test time? Leave-one-synthesizer-out splits test one factor at a time. They cannot separate position bias (PhonemeFake documents early-sequence bias), lexical polarity, and codec confounds, because these covary inside every existing dataset. The grammar subsumes leave-one-out as a special case, and it adds held-out combinations of factors that each appeared individually during training.
Why not just rewrite edits to be non-antonym? This fixes one known shortcut and leaves the unknown ones in place. Shortcut share across all eight factors tells us whether lexical polarity was the dominant leak or merely the visible one.
Proposed Experiments
The first decisive check validates the instrument. We take published-recipe detectors trained on AV-Deepfake1M, PartialSpoof and PartialEdit and compute their shortcut share on decorrelated grammar draws. We then test whether that share predicts their measured cross-dataset drops, such as XLSR-SLS going from 2.55% to 23.72% EER, or SLM-AT going from 97.51% to near 0% WordF1. If shortcut share does not rank these failures, the grammar is not capturing what breaks real detectors, and we stop to revise it.
The second check is compositional transfer in the style of held-out games. Training excludes specific factor combinations (for example, VoiceCraft in-filling at late positions under a codec pass over the whole utterance) and one synthesizer entirely. Evaluation covers those held-out combinations and the real PartialEdit, PartialEdit-Codec, AV-Deepfake1M audio, and HAD test sets, reporting WordF1, EER, and AP at IoU 0.5. The comparison is against equal-size correlated training.
The third check addresses the modality gap. We compare the linear-projector speech LLM, the aligned variant, and audio-only CFPRF, all trained on decorrelated data. We report each model's shortcut share and its WordF1 on edits whose text is unchanged (null edits) or neutral. Those subsets are where the transcript cannot help. Gains confined to antonym edits would indicate lexical leakage surviving alignment.
We also release the grammar and its generator. That way future evaluations can draw fresh instances rather than reuse a fixed test set, which is the property that made DyVal resistant to memorization.
References
- Can LLMs Help Localize Fake Words in Partially Fake Speech?
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A Practical Introduction to the Ludii General Game System
- AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset
- PartialEdit: Identifying Partial Deepfakes in the Era of Neural Speech Editing
- An Investigation of Spoofing Speech Detection Under Additive Noise and Reverberant Conditions
- Recall Residualisation: Decontaminating Foundation-Model Evaluation on Public Time-Series Benchmarks
- CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning
- Coarse-to-Fine Proposal Refinement Framework for Audio Temporal Forgery Detection and Localization
- PhonemeFake: Redefining Deepfake Realism with Language-Driven Segmental Manipulation and Adaptive Bilevel Detection