Honest Fusion Evaluation for Cross-Domain Deepfake Detection: Leakage Audits and Tail Robustness

Luke Collins, Chang-Tsun Li, Duc Thanh NguyenarXiv 2026 · arXiv

TL;DR

Most reported performance gains from complex "expressive" fusion methods in deepfake detection are actually artifacts of evaluation leakage. By auditing 22 datasets, this paper shows that simple calibrated averaging is just as effective for ranking, but the real power of expert fusion lies in rescuing the "tail"—the hardest videos that single models consistently miss.

What Problem Does This Paper Address?

Deepfakes are a moving target. A detector trained to find spatial blending artifacts in face swaps might completely fail to catch the temporal inconsistencies in a reenactment video. To solve this, researchers often combine multiple "frozen" expert models (e.g., one for texture, one for motion).

The prevailing wisdom in recent literature is that complex combination methods—like per-video routing (using a "gate" to pick the best expert for a specific video) or feature fusion (training a new head on top of concatenated features)—outperform simple score averaging. This paper investigates whether these gains are real or if they are simply "cheating" by exploiting subtle overlaps between training and testing data, a phenomenon known as evaluation leakage.

Methods

The authors built a massive testbed using four architecturally diverse frozen experts:

  1. DINOv3: A spatial expert.
  2. V-JEPA 2.1: A temporal expert.
  3. FREQ-DCT: A spectral (frequency-domain) expert.
  4. FSFM: A face-specific foundation model.

They compared several "combiners," ranging from simple averages to complex gradient-boosted routers. Crucially, they introduced a Corner Theorem, which mathematically proves that for ranking metrics like AUC, no "label-blind" router can ever beat a specific theoretical ceiling.

To audit for leakage, they tested across a "leakage ladder" of evaluation protocols:

  • Crossfit: Random splits (highly leaky).
  • LODO (Leave-One-Dataset-Out): Common, but leaky because "sibling" datasets often share the same source actors or pristine videos.
  • LOMO (Leave-One-Manipulation-Out): Leaky because the same dataset's "real" videos remain in training.
  • LOFO (Leave-One-Family-Out): The "honest" protocol, where entire families of related datasets are held out.
Visualizing the leakage ladder: how different protocols allow information to slip from training to testing.

Datasets

The study utilizes a comprehensive suite of 22 video deepfake test sets, including industry standards like FaceForensics++, DFDC, Celeb-DF, and newer sets like FakeAVCeleb and DeeperForensics. These are grouped into 12 families to ensure that "honest" evaluation truly tests generalization to unseen identities and manipulation techniques.

Key Insights and Ideas

  • The "Win" that Wasn't: Under the honest LOFO protocol, the supposed gains of complex routers and feature fusion virtually disappear. For example, a +2.6pp "win" for feature fusion was revealed to be a pure artifact of training-fold granularity.
  • The Corner Theorem: The authors prove that the optimal way to route scores requires knowing the test label (an "oracle"). Since real routers can't see labels, they struggle to beat a well-calibrated global average.
  • The Power of the Tail: While complex fusion doesn't help the average case (AUC), it is vital for the tail. These are the "hard" videos that every single expert gets wrong.
  • Diversity Law: Adding even a "weak" expert (like the spectral CNN) is valuable if it is diverse. It might not raise the average AUC much, but it can "rescue" videos that the stronger spatial/temporal models are blind to.
The feature fusion artifact: gains are significant under leaky manipulation folds but collapse to zero under honest family folds.

Key Results

  • Ranking Equivalence: Under honest evaluation (LOFO), all score-space combiners landed in a narrow 0.946–0.949 macro-AUC band. The complex "gate" actually performed slightly worse than simple averaging (-0.07pp).
  • Tail Robustness: Fusion significantly reduces the "All-Wrong" rate. As experts were added (from 1 to 4), the fraction of videos that every expert misclassified dropped from 15.2% to 2.3%.
  • Calibration is King: The only robust gain for combiners came from Inverse-Brier (INVB) weighting, which weights experts by their reliability. This outperformed the best single expert on worst-family AUC by +4.2pp.
Comparison of combiners under the honest LOFO protocol. Note how the "expressive" combiners fail to provide a meaningful boost over the simpler "stack" or "INVB" methods.

Limitations and Future Work

  • Frozen Experts Only: The findings apply to post hoc fusion of frozen models. The authors did not test end-to-end joint fine-tuning, which might behave differently.
  • Dataset Bias: The 22-dataset suite is currently skewed toward GAN and graphics-based fakes. Future work is needed to test these "honest" protocols against newer diffusion-based and audio-driven deepfakes.
  • Social Media Robustness: The impact of social media re-encoding (compression) on these fusion strategies remains an open question.

The paper concludes with a practical recipe for researchers: calibrate each expert in-fold, combine globally using Inverse-Brier weighting, evaluate using LOFO, and always report tail metrics.

Summary updated

Abstract

Cross-domain video deepfake detection increasingly uses frozen foundation-model experts, combined by calibrated score averaging, a per-video router picking one expert, or a learned head over concatenated features. Expressive routing and feature fusion are often reported to beat plain averaging. Across a 22-dataset frozen-expert testbed, we find those gains are largely evaluation artifacts. A corner theorem locates the gap routing aims to close: because AUC depends only on score ordering, the best any score-space router can do is an oracle reading the test label to pick each video's expert, a ceiling no realisable, label-blind router reaches. The theorem does not order real routers against a calibrated global average; the case against routing is empirical, and no router we test beats that average. Feature fusion reads raw features, falls outside the theorem, and is audited empirically. Reported gains trace to leakage: holding out one dataset leaks siblings sharing identities and reals (over-crediting routing), while holding out one manipulation keeps that dataset's reals in training (over-crediting feature fusion). A $2\times 2$ over fold and evaluation granularity pins the $+2.6$pp feature-fusion "win" to the fold alone: 12/12 significant under leaky folds vs. 0/12 under honest family-held-out folds, where both gains vanish. Fusion still pays off in the tail of hard videos and families: as experts are added, videos every expert misclassifies drop from 15.2% to 2.3%, and global calibrated fusion (InvB) beats the best single expert on worst-family AUC, selective risk (AURC), and low-false-positive recall, stably across protocols. Practitioners should calibrate and combine globally, hold out whole dataset families, and measure tail performance. For score-space routing under ranking metrics, honest evaluation and expert diversity matter more than combiner complexity.
BibTeX
@misc{collins2026honest,
  author = {Luke Collins and Chang-Tsun Li and Duc Thanh Nguyen},
  title = {{Honest Fusion Evaluation for Cross-Domain Deepfake Detection: Leakage Audits and Tail Robustness}},
  year = {2026},
  eprint = {2611.14856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14856},
}
Figure