Explaining Away Transcription Errors in Short-Video Misinformation Detection

Short-video fact-checking models read the speech track through an automatic speech recognizer, a system that turns audio into text and makes mistakes. When the recognizer mishears "guitar" as "gucci", the transcript no longer matches what the video shows. To a detector trained to look for disagreement between what is said and what is shown, that error looks exactly like a lie. The opposite fix is just as risky. Recent work corrects transcripts by pulling them toward what the video depicts. That assumes the video is honest, but the typical "cheapfake" pairs real footage with a false spoken claim. Correcting toward the video would erase the very mismatch that reveals the deception.

We propose a detector that treats a speech-video mismatch as evidence of deception only when no acoustically plausible reading of the speech fits the video. The method has three steps. First, it marks the parts of the transcript that carry a checkable claim, namely the arguments of the main predicates. Second, it keeps several alternative transcriptions for each of those parts, along with the recognizer's confidence. Third, it asks whether any reasonably likely alternative agrees with the frames. If one does, the mismatch is attributed to the recognizer. If none does, the mismatch counts as evidence of deception. The audio decides which corrections are allowed, so the video cannot rewrite a confidently spoken lie, and recognition noise cannot pass as deception.

Core Claim

Current short-video detectors misread recognition noise as deceptive mismatch, and the cost falls on authentic videos with poor audio. Such videos are disproportionately amateur, colloquial and user-generated. This is the "ugly truth" bias noted by FakingRecipe, seen through the speech channel. We claim that limiting the attribution of a mismatch to transcription error to acoustically plausible alternatives on claim-bearing words will do two things. It will cut false "fake" verdicts on real videos whose transcripts have low confidence. It will also generalize better to held-out topic domains, because recognition quality varies by topic and recording style and therefore acts as a domain-specific shortcut. We do not expect large gains in overall accuracy. The claim stands or falls on the low-confidence slice and on held-out domains.

Where This Sits

Two papers motivate the gap from opposite sides. Visual Information Matters for ASR Error Correction prepends frame captions to noisy transcripts so that a sequence-to-sequence model can fix homophones, reporting a 1.2% absolute word-error-rate reduction. It assumes the frames are truthful. We reuse its insight that vision can disambiguate speech, but we forbid any correction that the audio does not support. Inconsistency Matters separates cross-modal inconsistency from inconsistency with external knowledge, but it works on clean written posts. We add a third source of inconsistency, transcription error, and separate it from the other two.

On the detection side, three systems supply the strongest competitors but ingest transcripts as flat, uniformly trusted text. HyperClaim links claim tokens, transcript segments and frame windows in a temporal hypergraph, a graph whose edges can join many nodes at once. DOCTOR removes domain-specific artifacts from fused features with a diffusion model to generalize across FakeSV's nine topics. FactGuard runs an agent that retrieves web evidence and inspects video clips. In FactGuard, a misheard entity also corrupts the search query. None of the three represents recognition uncertainty.

From the uncertainty literature we reuse the principle of passing confidence downstream rather than correcting first. Incorporating Speech Recognition Confidence into Discriminative NER showed that confidence features raise named-entity precision on Japanese speech from 70.67% to 78.13%.

The syntactic component borrows from two papers but plays a narrower role than either. Syntax Matters restricts attention to dependency ancestors on noisy spoken transcripts, raising TREC6 accuracy from 84.1% to 86.4%. Its gating against parser errors is a design we reuse. Opinion-based Relational Pivoting found that dependency paths are 94–97% shared across review domains while aspect vocabulary overlaps only about 20.5%. That suggests structural roles transfer across topics where vocabulary does not.

One paper constrains us. The Aligned Multimodal Movie Treebank shows standard parsers degrade sharply even on scripted film dialogue. Full trees over colloquial recognized speech are therefore unreliable. We use syntax only to locate predicate heads and their arguments, the "slots" where a factual claim lives, and we test whether that localization helps at all.

Research Program

The pipeline begins by re-transcribing the audio of each video rather than using the dataset-supplied transcript. We use an open multilingual recognizer with beam search, plus a second recognizer, to obtain an N-best list (the top-scoring alternative transcriptions) with token posteriors. A shallow dependency parse of the top transcription identifies claim slots: the subjects, objects and obliques of main predicates, plus named entities and numbers. Each slot is aligned to its time span in the audio and to the corresponding alternatives in the N-best list.

For a slot $a$ with alternative readings $h_1,\dots,h_K$ and recognizer posteriors $p_1,\dots,p_K$, let $s(h_k, V)$ be a text-to-video compatibility score in $[0,1]$. This score is computed with a CLIP-style image-text encoder over frames in the slot's time window, together with any on-screen text. The deception evidence for the slot is

$$ d(a) = 1 - \max_{k:\, p_k \ge \tau} s(h_k, V), $$

where $\tau$ is an acoustic plausibility threshold tuned on validation data. A slot is explained away when its top reading mismatches the video but some other plausible reading fits. It is flagged when even the best plausible reading fails to fit. When the top reading has high confidence, it is the only reading above $\tau$, so a confidently spoken false claim can never be corrected toward the footage.

The per-slot scores $d(a)$, the explained-away mass, and slot confidences are pooled. They are added as an extra readout to a fixed backbone detector. We use a reimplementation of the HyperClaim discrepancy readout, and a simpler cross-attention detector as a second backbone. A gate in the style of the Dual Context Fusion Module of Syntax Matters lets the model fall back to unstructured features when the parse looks broken.

Training adds counterfactual audio degradation. We apply compression, background noise and reduced bitrate to real videos, keep their labels, and re-transcribe them. This teaches the detector that worse audio is not evidence of fakeness.

Baselines

The naive reference point is the backbone detector with dataset-supplied transcripts and no uncertainty handling. Its false-fake rate on low-confidence real videos defines the problem. Because audio-quality effects are invisible in aggregate accuracy, we report it per slice.

The strongest existing method is HyperClaim. On the chronological split it reports 83.7% accuracy on FakeSV, 82.0% on FakeTT and 87.3% on FakeVV. On FakeSV, other papers report higher numbers under different or unspecified protocols. FakingRecipe reports 85.35% accuracy and DOCTOR reports 84.91% accuracy with 84.70% F1. These are not directly comparable to HyperClaim's temporal-split figure, and the file does not state DOCTOR's split. We will therefore retrain HyperClaim and DOCTOR ourselves on identical splits, with the same frozen video and text encoders, the same transcripts, and the same hyperparameter search budget, rather than quoting their numbers.

For domain generalization, the DOCTOR reproduction is the main competitor. We found no per-held-out-domain result on Lacuna that we could quote for it.

Beating these baselines means two things on the temporal and leave-one-domain-out splits. Overall accuracy must not be significantly lower. The false-fake rate on the lowest-confidence tercile of real videos must be lower, with confidence intervals from five seeds that exclude zero difference.

Three ablations isolate the key ingredient. In the first, confidence is fed as a plain per-token feature to the backbone, with no alternatives and no slots. In the second, alternatives and $d(a)$ are used, but slots are all content words rather than parser-selected arguments. In the third, $\tau$ is set to zero, which allows unconstrained vision-guided correction and recreates the risk identified by Kumar et al.

Why not just…?

Why not just correct the transcript first with a vision-guided corrector and run any detector? The worry is concrete. A corrector trained to make speech agree with frames, as in Visual Information Matters for ASR Error Correction, will rewrite claims in cheapfakes where the footage is real but unrelated. We test this directly. We build a cheapfake probe set from fake FakeTT and FakeSV videos whose claims contradict the visuals, and measure how often the corrector changes claim slots toward the visual content. The $\tau = 0$ ablation is this pipeline inside our own architecture. If the corrector does not in fact erase deceptive mismatches, our constraint is unnecessary and we will report that.

Why not just add recognizer confidence as an input feature and let the network learn the rest? This is the most serious threat. It worked for NER in Incorporating Speech Recognition Confidence and costs almost nothing. Our argument is that confidence alone says a word might be wrong, but not whether the right word would fit the video. Only alternatives scored against the frames can separate "misheard" from "lied". If the confidence-feature ablation matches the full model on the low-confidence slice, the alternative-checking mechanism is not needed. We would then publish the cheaper method as the finding.

Why not just train with audio-noise augmentation, or use a better recognizer? Augmentation with preserved labels is already part of our method. We run it alone as a baseline, because it may remove the "bad audio means fake" shortcut by itself. A stronger recognizer lowers the error rate but cannot remove homophone ambiguity in noisy user audio. We test this by running all baselines on the best available transcripts and checking whether the gap on the low-confidence slice remains.

Why not just prompt a frontier multimodal model with the raw video? The evidence suggests this falls short. HyperClaim reports GPT-4o at 56.0% accuracy on FakeVV, and FakingRecipe documents GPT-4V labeling TikTok content as fake regardless of veracity. We include a zero-shot multimodal model given N-best transcripts as a reference row. We do not treat it as the main competitor.

Why use syntax at all, given that parsers fail on colloquial speech? The skepticism is fair, given the parser degradation documented on AMMT. Syntax enters only as a slot selector. The content-word ablation decides whether it earns its place. If the two are indistinguishable, we drop the parser.

Proposed Experiments

The main experiments use FakeSV and FakeTT with three protocols. The first is the FactGuard chronological split, in which the latest 15% of videos is the test set. The second is leave-one-domain-out over the nine shared topics, following the cross-domain setting DOCTOR targets. The third is a cross-confidence split: train on high-confidence videos, test on low-confidence ones. FakeVV serves as a check that entity-replacement detection is not harmed.

Metrics are accuracy and macro-F1. Alongside them we report the false-fake rate on real videos, sliced by recognizer confidence and by audio degradation level. We also report the correlation between the detector's fake score and transcript confidence among real videos, which should shrink toward zero.

Two component-level checks validate the mechanism separately. On Visual-ASR-EC, we test whether "explained away" decisions identify true recognition errors, measured by precision and recall against gold transcripts. On the cheapfake probe set, we test whether any deceptive slot is wrongly explained away.

The decisive check is a counterfactual audio test on held-out real videos. We degrade the audio, re-transcribe it, and compare the resulting changes in verdicts across all systems. If our model flips real videos to "fake" significantly less often than both HyperClaim and the confidence-feature ablation, without missing more fakes, the claim holds. If not, the proposal reduces to the simpler confidence baseline.