Learning Where Speech Was Forged from Transcript Mismatches

Detectors that find where a recording was tampered with, rather than just whether, need training labels that mark every forged frame. Today those labels come from the forgery generator itself, so each detector learns only the splicing habits of the few generators its dataset used. The known result is collapse on unfamiliar ones. A cheaper and more varied label source already exists. Take a manipulated clip and a reference record of what was actually said, such as an original transcript or the untouched source recording. Transcribe the clip, find the words that differ from the reference, and time-align them to the audio. The result is a frame-level "silver" label, obtained the same way relation-extraction systems obtain labels by matching text against a database.

These labels are noisy, and the noise has a known shape. Speech recognition errors create words that "differ" without being forged. Homophone-like edits get auto-corrected and missed. Forced alignment shifts word edges by tens of milliseconds. Crossfaded splices have no single true edge. This note proposes to treat forgery localization as learning from such distant supervision. Transcript mismatch is a noisy proxy for forgery, just as entity co-occurrence is a noisy proxy for a relation. The model separates two kinds of label error that need different fixes: flipped word-level labels, and jittered boundaries. The transcript is used only to make labels, never as a model input, which keeps the detector from learning text-based shortcuts.

Core Claim

A frame-level localizer trained on transcript-diff silver labels, with an explicit two-part noise model, will close most of the gap to the same architecture trained on generator-provided gold labels. The first part estimates a word-level flip model for recognizer false positives and missed edits. The second marginalizes over latent boundary offsets for alignment jitter and crossfade width. Because silver labels can be produced for any generator, including closed commercial voice cloners and inpainting systems that do not expose splice points, the extra generator diversity should improve out-of-domain localization beyond what gold-only training achieves. Out-of-domain localization is currently the field's worst failure.

The new element is not weak supervision in general. It is a specific, auditable noise channel between a text-level signal and a frame-level target, and a learning rule matched to that channel.

Why This Gap Is Real

Three findings motivate the proposal.

The first is how badly localizers fail out of domain. Robust Localization of Partially Fake Speech re-evaluates the two-stage proposal model CFPRF, which scores frames and then refines segment boundaries. Its equal error rate goes from 7.61% on PartialSpoof to 43.25% on LlamaPartialSpoof. Equal error rate is the rate at which false alarms equal misses, computed without fixing a threshold. The same paper shows that this metric hides even worse behavior at a fixed operating threshold.

The second is where that failure comes from. LlamaPartialSpoof builds its fakes with an LLM that rewrites scripts, six text-to-speech systems, alignment boundaries from the Montreal Forced Aligner, and five crossfade functions. Its best PartialSpoof-trained baseline reaches only 24.49% equal error rate, and other models reach up to 57.18%. The authors attribute this to overfitting to one splicing implementation. That diagnosis points to training diversity, which gold labels make expensive.

The third is a limit on naive scaling. Luong et al. found that adding fully fake or fully real utterances hurt localization. Only partially fake utterances with boundary contrast helped. Silver labeling produces exactly that kind of data.

Existing ways around gold labels either discard temporal information or assume their auxiliary labels are clean. The weakly supervised methods WMMT and MDP use only a clip-level "real or fake" label. They add a loss that rewards large feature jumps between adjacent segments. This gives 84.57% average precision at a temporal overlap threshold of 0.5 ([email protected]) on LAV-DF for MDP, still well short of full supervision. Both assume the clip label is correct, and both struggle at strict overlap thresholds.

LOCO goes further by turning its own first-stage proposals into frame pseudo-labels. This invites the confirmation bias that the distant-supervision literature warns about, because the model's early mistakes become its later training targets.

NE-PADD is the closest existing work. It aligns Whisper transcripts and Stanza named-entity tags to frames and uses them as an attention prior. It treats these text-derived labels as exact, and it feeds them to the model as input.

Feeding text in is the danger that Can LLMs Help Localize Fake Words in Partially Fake Speech? exposes. Its speech language model reaches 97.51% word-level F1 on AV-Deepfake1M. It falls to nearly 0% on PartialEdit, because it learned that sentiment-flipped words are fake. Our design uses text strictly as a supervisor, so no text path exists at test time.

Research Program

Silver-label pipeline. For each partially fake clip with an available reference, we transcribe both the clip and the reference with Whisper. A reference is the bona fide source clip or the dataset's original script. We compute a word-level edit alignment between the two transcripts. Words present only in the clip become candidate forged words, and the forced aligner assigns them start and end times.

Noise audit. On datasets that release gold segments, we measure four quantities directly. The first is the word false-positive rate, meaning recognizer disagreements on authentic words. The second is the missed-edit rate. The third is the distribution of boundary offsets between aligned and gold edges, broken down by crossfade type where LlamaPartialSpoof records it. The fourth is the rate of gross misalignment. This audit is a standalone deliverable, since the field has no measurement of how far text-derived temporal labels sit from generator truth. It also tells us whether a structured noise model is needed at all. If offsets turn out to be small, symmetric and unimodal, simple boundary smoothing should suffice, and we would report that.

Learning rule. Following the decoupling idea of DesERT, the model has two heads. DesERT separates "is there an entity here" from "what type is it" so that the dominant background class does not drown out rare classes. Our analogue is a word-level head that predicts whether a word span is forged, and a frame-level head that predicts the exact edge location inside and around that span.

The word head is trained through a two-by-two flip matrix. The matrix is estimated from a small gold anchor set of 5% of training clips. We avoid going lower because Nagesh et al. report teacher-student consistency collapsing at around 1% labeled data.

For the boundary head, let $\tilde{s}$ be the silver onset frame and let $\delta$ be an unknown offset with a learned distribution $p_\phi(\delta)$ over a window of $\pm K$ frames. The frame head's log-likelihood of a forged segment starting at frame $s$ is $\log p_\theta(s \mid x)$, where $x$ is the audio. We train on the marginal

$$\mathcal{L}_{\text{bnd}} = -\log \sum_{\delta=-K}^{K} p_\phi(\delta \mid c)\, p_\theta(\tilde{s}+\delta \mid x),$$

with the same form for the offset edge. Here $c$ is a cheap context feature, such as aligner confidence or the energy slope around the edge, that allows the offset distribution to be skewed and to vary across cases.

This marginalization differs from label smoothing in one important way. The model can place all its mass on the offset that the audio supports. Smoothing instead forces the model to spread its mass. The marginal also absorbs gross misalignments without distorting the predicted edge.

For the backbone, we use the BAM architecture of Enhancing Partially Spoofed Audio Localization with Boundary-aware Attention Mechanism. BAM predicts boundaries and blocks attention across them. Its boundary head is the natural place to attach $\mathcal{L}_{\text{bnd}}$.

Diversity experiment. We then generate new partially fake speech with closed or black-box systems, including inpainting-style editing of the kind used in PartialEdit and the commercial voice cloners listed for LlamaPartialSpoof. We label this data only through the silver pipeline and test whether adding it improves out-of-domain localization.

Baselines

Naive silver. The reference point is the identical BAM model trained on silver labels as if they were gold. The gap between this and our method measures what the noise model contributes. The gap between this and gold training measures how damaging the noise is.

Gold-label upper bound. This is the same model trained on generator labels. For published context we use reported numbers:

We will re-run the gold models ourselves rather than compare against these numbers directly. Every model gets the same encoder, the same training clips, and the same hyperparameter search budget, which is equal trials selected on a held-out gold development split.

Weak supervision. The strongest method that avoids frame labels is MDP, at 84.57 [email protected] on LAV-DF, with WMMT reporting 73.27 average mAP there. On audio we include LOCO. For LOCO the file reports only relative gains, so we have no absolute number and will run it.

Generic noise handling. These are the alternatives a practitioner would try first:

  • Gaussian boundary label smoothing with a tuned width.
  • An ignore-zone of $\pm K$ frames around silver edges.
  • DesERT-style dual co-guessing, in which two peer networks keep a silver word only when both agree confidently.

Ablations. We remove the offset marginal, remove the word flip model, and separately add the transcript as a model input. The last ablation directly tests whether a text path brings back the lexical shortcut.

Beating the baselines. In-domain, our method must beat naive silver and every generic noise handler at [email protected] and [email protected], since strict overlap is where boundary noise should matter most. It must also reach at least MDP-level [email protected] on LAV-DF. Out-of-domain, it must beat gold-only training on fixed-threshold F1.

Why not just…?

Why not just use clip-level weak supervision, as MDP and LOCO do? Clip labels give no information about where the edit is. Weak methods lose most at strict overlap: MDP reports 84.57 [email protected] against 96–98 for full supervision, and it names boundary precision as an open problem. Silver labels provide approximate locations at the same annotation cost, provided a reference exists. The comparison against MDP and LOCO at [email protected] and [email protected] settles the question.

Why not just smooth the labels near the boundaries? This is the most serious threat. If the audit finds symmetric, narrow and unimodal offsets, smoothing may match our method, and we will say so. We expect otherwise, for two reasons. Forced aligners are likely to shift edges in a consistent direction, and crossfades widen the ambiguous region by varying amounts, so a single fixed smoothing width is wrong for most edges. The tuned-smoothing baseline, broken down by crossfade type, decides this.

Why not just generate your own fakes and log exact splice points? For generators you run yourself, this gives gold labels, and it is what current datasets do. It does not reach closed services, or inpainting tools whose regenerated region can extend beyond the requested words. It also cannot label third-party media paired with a reference. An equal-diversity control therefore matters. If gold labels from the same set of generators match silver-plus-noise-model on out-of-domain tests, our contribution reduces to cheaper labeling rather than better learning. We will report the result either way.

Why not just transcribe the clip and have a text model flag odd words? The Speech-LLM study shows that text-driven localization collapses to near-zero word F1 when editing style changes. LlamaPartialSpoof uses an LLM specifically to make edits fluent and plausible. Our ablation that adds a transcript input measures the same shortcut within our own setup.

Proposed Experiments

Training data. We train on PartialSpoof, LAV-DF (audio track), and AV-Deepfake1M, each with gold and with silver labels. We add the black-box-generated silver data in the diversity condition.

Test data. We test on the in-domain test splits. Out-of-domain, we test on LlamaPartialSpoof and PartialEdit, with HAD as a cross-lingual check. Note that Whisper's accuracy on Chinese will shift the silver noise profile for HAD.

Metrics. Following Luong et al., we report frame equal error rate together with precision and F1 at a threshold fixed on the in-domain development set. We also report AP at overlap thresholds 0.5, 0.75 and 0.95, and word-level F1.

Decisive checks. Three results decide the proposal.

  1. Recovery of gold performance. Our method should recover at least most of the gold-versus-naive-silver gap at [email protected], and it should beat tuned smoothing and co-guessing there.
  2. Robustness under misalignment. In a controlled-corruption study, we inject increasing rates of gross misalignment into gold labels. Our method should degrade more slowly than smoothing.
  3. Out-of-domain gain from diversity. With the diversity data added, fixed-threshold F1 on LlamaPartialSpoof should exceed gold-only training.

If the first check fails, the noise model is wrong. If only the third fails, silver labeling does not pay for itself in generalization, and the audit and noise model stand as the contribution.