Forensic Drafts from the Top Bits: Speculative Decoding for Quantized Audio-Visual Deepfake Explainers

Large multimodal language models can do something specialized deepfake detectors cannot. They can say which stream was forged, when, and why, and they are less tied to the dataset artifacts that trip up small classifiers. They are also too slow and too large to run on live video. Two standard remedies exist. Quantization stores weights in very few bits. Speculative decoding lets a cheap "draft" model propose several tokens that the full model then checks in one parallel pass, which gives exactly the full model's output at lower latency. This proposal combines the two for audio-visual forgery analysis. The drafter is the 2-bit "top half" of the same 4-bit model that verifies it, so it costs no extra memory. The design question is how to choose the few weights that stay at higher precision in that drafter.

The core observation is that quantization and cheap drafting fail on the same tokens. Aggressive quantization keeps fluent text but destroys the model's ability to reason across modalities (An empirical study of LLaMA3 quantization). Cross-modal forensics depends entirely on binding sound to lips and faces (Explicit Correlation Learning; AuViRe). A 2-bit drafter protected in the usual way, by preserving weights that matter for text perplexity, should therefore be accepted on the connective prose of an explanation but rejected on the verdict, the manipulated-modality label, and the timestamps. We propose to choose the protected weights by how much they preserve drafter–verifier agreement on exactly those evidence tokens, and to test whether this closes the gap.

Core Claim

Consider a forensic model that writes a structured explanation for a clip. The output has a verdict, which stream was manipulated, a time segment, and a short rationale. Most of its tokens are connective text that any fluent model predicts well. A small fraction are evidence tokens whose correct value depends on cross-modal binding. For a draft distribution $q$ and verifier distribution $p$ at a given position, the standard lossless acceptance rule accepts a drafted token with probability $\min(1, p/q)$ (speculative decoding survey). The expected acceptance at that position is therefore

$$\alpha = \sum_x \min\big(p(x), q(x)\big) = 1 - \mathrm{TV}(p, q),$$

where TV is total variation distance. The expected acceptance is simply one minus how differently the two models distribute probability at that position.

Our claim has two parts. First, a 2-bit drafter calibrated on text will show a large acceptance gap between connective tokens and evidence tokens. The gap will be largest on partial fakes, such as real audio with a swapped face, where binding matters most. Second, at the same bit budget, choosing the protected full-precision weights to minimize TV on evidence tokens will recover most of that gap. The result would be higher wall-clock speedup with output identical to the 4-bit verifier.

The novelty is not "QSpec for video." It is using the verifier's own agreement on cross-modal evidence tokens as the objective for mixed-precision allocation in a self-speculative drafter. A side contribution is a measurement: per-token rejection maps become a direct view of where quantization breaks cross-modal binding, which accuracy-only studies cannot provide.

Research Program

Verifier model. We fine-tune an open audio-visual ("omni") multimodal model to emit structured forensic explanations. The training targets are generated from labels in FakeAVCeleb, whose four real/fake audio × video categories give the manipulated-modality field (FakeAVCeleb). AV-Deepfake1M and LAV-DF supply segment timestamps. This fine-tuning is necessary: zero-shot omni models reach only 0.618 mean AUC on MADBench (MADBench).

Shortcut control. Leading silence is trimmed from all clips. Supervised detectors on these datasets exploit a 25–30 ms silence artifact, and removing it drops their AUC from 99.2% to 54.9% (AVH-Align). The verifier is then quantized to W4A16 with a rotation-based method in the spirit of QuIP.

Drafter. The drafter reads only the two most significant bit-planes of each 4-bit weight, so each draft step loads roughly half the weight bytes. It also keeps a small protected set of weights at full precision, about 0.4% following the budget CherryQ showed suffices for text at 2–3 bits.

Shared cache. Following QSpec, the drafter reuses the verifier's high-precision key-value cache: the stored intermediate states of the audio-visual prefix and of already-verified tokens. The drafter therefore never re-encodes video or audio. This matters because re-encoding visual context in a separate drafter can cost up to 25× the drafting time itself (multimodal diffusion drafting diagnosis). It also means the drafter can see cross-modal information that a modality-blind drafter cannot.

Choosing the protected weights. We score each weight by the diagonal Fisher sensitivity of the TV objective above, computed on evidence tokens from a few hundred calibration clips that are balanced across real, full-fake, and partial-fake categories. The top-scoring weights are protected. This replaces CherryQ's objective, Fisher sensitivity of language-modeling loss on text, with verifier agreement on the tokens forensics cares about.

Analysis. We will produce three diagnostics:

  • per-layer and per-module overlap between text-derived and evidence-derived protected sets, which directly tests whether cross-modal binding lives in different weights;
  • acceptance broken down by token type and forgery category;
  • a standalone evaluation of the unverified 2-bit drafter as a detector, to measure how much raw forensic ability quantization destroys and where.

Baselines

Naive reference. The W4A16 verifier decoding one token at a time. Its detection metrics define ground truth, because every speculative variant must reproduce them exactly. Its per-clip latency defines speedup 1.0×. As an external latency point, the knowledge-guided visual forensic model runs at about 1.2 seconds per frame (Unlocking LVLMs for deepfake detection).

Strongest existing drafter design: QSpec's scheme applied to this verifier. Drafting uses 4-bit weights with 4-bit activations; verification uses 16-bit activations; the cache is overwritten after verification. QSpec reports 1.32× to 1.64× speedup on text Llama models (3B–13B) with MATH accuracy restored to the ~24% W4A16 level (QSpec). It is the closest method: single model, zero extra memory, quantization-based drafting. It is run on identical clips, hardware, batch sizes (1 and 8), and draft lengths. We beat it if our drafter gives higher mean accepted tokens per verification and higher wall-clock speedup at equal output.

Separate small drafter. A small text-only drafter fine-tuned on the explanation corpus, following the image-blind drafter that reached up to 2.37× memory-bound speedup on LLaVA-7B (On Speculative Decoding for MLLMs). It receives the same fine-tuning data and GPU-hour budget as our calibration plus any drafter tuning. There is no reported result on Lacuna for any speculative decoding method on an audio-visual forensic model, so all speedup comparisons are produced in this study.

We omit hierarchical tree verification (HierSpec, 2.78× on W4A16 Llama-3-70B) as a primary baseline. It needs a third model tier. We use linear drafts, which avoid the tree-verification cost HierSpec identified. We do report HierSpec's number as the text-domain ceiling.

Key ablation. The 2-bit top-bit drafter is held fixed at the same protected-weight budget, and only the rule for choosing protected weights changes:

  • random selection;
  • text Fisher on C4-style data (CherryQ-style);
  • multimodal Fisher on all output tokens;
  • evidence-token agreement (ours).

Beating text Fisher specifically on evidence-token acceptance, and on partial-fake clips, is the result that supports the core claim.

Detection-accuracy references. Specialized detectors are reported for context, not as speed competitors:

  • MRDF: 92.43% AUC on FakeAVCeleb (MRDF);
  • AVH-Align: 85.24% AUC on AV-Deepfake1M;
  • AuViRe: 99.8 AUC on AV-Deepfake1M (AuViRe).

Why not just…?

Why not just deploy a small specialized detector like AuViRe or AVH-Align? For video-level AUC on in-distribution benchmarks they are hard to beat, and we will say so. They return a single score without modality attribution or rationale. They are also brittle: AVH-Align falls to about 0.52 AUC when only ambient audio is forged (MADBench), and supervised models collapse once the silence shortcut is removed. Our goal is to make the explainable model affordable, not to beat these detectors on AUC. The MADBench and partial-fake stratifications will show whether the explainer's added coverage justifies its cost.

Why not just use a modality-blind small drafter, since explanations are mostly connective text? This is the most serious threat. Evidence tokens may be few enough that rejecting them barely hurts aggregate speedup. The per-token-type acceptance breakdown and wall-clock comparison against the text-only drafter settle it directly. If the blind drafter matches us within noise, we will report that cross-modal-aware drafting is unnecessary for this output format and narrow the contribution to the diagnostic findings. We expect it to lose because each evidence-token rejection truncates a draft block, and timestamped explanations contain several such tokens.

Why not just recover the 2-bit model with QLoRA and skip verification? Fine-tuning heavily quantized models on small corpora degraded pre-trained ability in LLaMA3 (LLaMA3 quantization study). An unverified model would also give no guarantee that its verdict matches the full model's, which matters in forensic use. As a fair test, we include a variant where the drafter gets a small adapter trained to match the verifier's evidence-token distributions under the same compute budget. If it beats weight protection on acceptance, that is a legitimate finding and we will report it.

Why not just protect weights with standard text-calibrated saliency? That is exactly the key ablation. If text and evidence protected sets overlap heavily and give equal acceptance, then cross-modal binding does not need its own precision budget, and the proposal's central claim is false.

Proposed Experiments

Data and metrics. Experiments cover FakeAVCeleb, AV-Deepfake1M, and LAV-DF, all with leading silence trimmed. MADBench is used for environmental-audio forgeries.

  • Losslessness: verdict AUC and segment [email protected] / [email protected] must be identical across all speculative variants. Any difference indicates a bug, not a tradeoff.
  • Efficiency: mean accepted tokens, acceptance split by connective versus evidence tokens, and wall-clock speedup at batch sizes 1 and 8.
  • Stratification: by the four FakeAVCeleb categories, with partial fakes (real audio, fake video) singled out because MRDF showed fused detectors misclassify them at 30.0% before regularization.

Decisive check 1: does the premise hold? Under a text-calibrated 2-bit drafter, evidence-token acceptance must fall clearly below connective-token acceptance, with the largest drop on partial fakes. If there is no such gap, the proposal reduces to transferring QSpec to multimodal inputs.

Decisive check 2: does evidence-token calibration help? At a fixed protected budget, evidence-agreement calibration must raise evidence-token acceptance and end-to-end speedup over text Fisher. It must also beat QSpec-style drafting and the blind drafter on wall-clock speedup.

Decisive check 3: are the protected sets different? The overlap analysis should show that the evidence-derived set concentrates in the layers where audio and visual tokens interact, rather than duplicating the text-derived set.

Diagnostic. The unverified 2-bit drafter's standalone AUC on partial fakes, compared with its fluency, quantifies the silent failure mode that motivates verification. A model that writes confident, fluent explanations while missing a swapped face should never be deployed unverified.