TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection

Xinqi Xiong, Prakrut Patel, Qingyuan Fan, Amisha Wadhwa, Sarathy Selvam, Xiao Guo, Luchao Qi, Xiaoming Liu, Roni Sengupta

TL;DR

This paper introduces TalkingHeadBench, a new, meticulously curated benchmark for detecting sophisticated talking-head deepfakes. It highlights that current state-of-the-art detectors, while performing well on older datasets, struggle significantly with modern, high-quality deepfakes, especially when identity or generation methods change, revealing a critical need for more robust detection models.

What Problem Does This Paper Address?

The rapid advancement of AI-generated talking-head videos has led to increasingly realistic deepfakes, posing significant risks in areas like misinformation and social engineering. Existing benchmarks for deepfake detection often rely on older generation techniques and contain many low-quality samples, making them insufficient for evaluating detectors against the latest, more convincing deepfakes. This gap means we lack reliable ways to measure how well current detection methods can generalize to new, sophisticated deepfake generation techniques and variations in identity, which is crucial for developing effective countermeasures.

Methods

The authors introduce TalkingHeadBench, a new benchmark dataset and evaluation framework. The dataset was created by:

  • Source Data: Using images from FFHQ and audio/video from CelebV-HQ.
  • Generation: Employing six state-of-the-art open-source talking-head generators (e.g., Hallo, AniPortrait, LivePortrait) and two additional emerging generators (MAGI-1, Hallo3) for generalization testing.
  • Curation: A rigorous five-stage expert-led curation process that removed over 60% of generated videos to ensure only high-quality, realistic samples remained, making the dataset more challenging.
  • Evaluation Protocols: Three protocols were designed to assess detector robustness:
  • Protocol 1 (Identity Shift): Tests generalization to unseen identities with familiar generators.
  • Protocol 2 (Generator Shift): Tests generalization to unseen generators with familiar identities.
  • Protocol 3 (Identity & Generator Shift): Tests generalization to both unseen identities and generators, simulating the most challenging real-world scenario.
  • Explainability: Grad-CAM was used to visualize detector attention, helping to understand why certain deepfakes are harder to detect and identifying common failure modes.

Datasets

  • Training/Evaluation Data: The newly introduced TalkingHeadBench dataset, comprising 2,994 high-quality talking-head deepfake videos and 2,312 real videos, curated from eight academic and commercial generators.
  • Existing Datasets (for context/comparison): FaceForensics++ [34] and CelebV-HQ [53] were used for real videos in evaluation splits to ensure no identity overlap with training data.

Key Insights and Ideas

  • The "Realism Gap": Modern talking-head deepfakes are significantly harder to detect than those generated by older methods, as evidenced by the performance drop of SOTA detectors on TalkingHeadBench compared to older benchmarks like FaceForensics++.
  • Generator-Specific Challenges: Not all deepfake generators are equally detectable. EMOPortraits [12] proved particularly challenging for most detectors, even under strict thresholds.
  • Shift in Detector Focus: As deepfake realism increases, detectors tend to shift their attention from facial features to background artifacts, indicating a potential vulnerability and a reliance on less robust cues.
  • Generalization is Key: Identity and generator shifts pose significant challenges. Models trained on one set of generators or identities struggle when presented with new ones, highlighting the need for more generalized detection capabilities.
  • Threshold Matters: Standard metrics like AUC can mask poor performance at critical low false positive rates (e.g., TPR@FPR=0.1%), which are essential for practical deployment.

Key Results

  • Performance Drop: State-of-the-art detectors that achieve near-perfect accuracy on older datasets like FaceForensics++ show a significant performance degradation on TalkingHeadBench, particularly under Protocol 3 (Identity & Generator Shift). For example, TALL's average T1 score dropped from 0.99 in P1 to 0.87 in P3.
  • Best Performers: TALL [49] and DeepFake-Adapter [39] emerged as the most robust detectors, maintaining relatively high performance across protocols and generators, though still showing degradation.
  • EMOPortraits Difficulty: EMOPortraits [12] was identified as the most challenging generator, with average T1 scores dropping to 0.44 across detectors.
  • Grad-CAM Insights: Visualizations showed that detectors sometimes focus on background cues, especially for generators like Hallo2 and EMOPortraits, suggesting a potential over-reliance on non-facial artifacts.
  • Success Case: TALL demonstrated effective generalization on some EMOPortraits videos by focusing on the neck region, indicating it could learn from subtle artifacts.

Limitations and Future Work

  • Limitations of SOTA Detectors: Even the best models struggle to achieve consistent excellence across all generators and protocols, particularly under combined identity and generator shifts. Detectors still show a tendency to rely on background artifacts.
  • Future Work:
    • Improved Generalization: Develop detectors that can generalize uniformly across diverse generation methods and handle subtle artifacts without overfitting to background cues.
  • Evolving Benchmark: The authors plan to evolve TalkingHeadBench into a dynamic platform with APIs for generator testing, dual leaderboards, and regular updates to keep pace with generative model advancements.
  • Fairness Audits: Conduct regular fairness audits to address potential demographic imbalances inherited from source datasets.
  • Target Performance: Future research should aim for >99% T1 and T0.1 scores across all generators and protocols.

Summary updated

Abstract

The rapid advancement of talking-head deepfake generation fueled by advanced generative models has elevated the realism of synthetic videos to a level that poses substantial risks in domains such as media, politics, and finance. However, current benchmarks for deepfake talking-head detection fail to reflect this progress, relying on outdated generators and offering limited insight into model robustness and generalization. We introduce TalkingHeadBench, a comprehensive multi-model multi-generator benchmark and curated dataset designed to evaluate the performance of state-of-the-art detectors on the most advanced generators. Our dataset includes deepfakes synthesized by leading academic and commercial models and features carefully constructed protocols to assess generalization under distribution shifts in identity and generator characteristics. We benchmark a diverse set of existing detection methods, including CNNs, vision transformers, and temporal models, and analyze their robustness and generalization capabilities. In addition, we provide error analysis using Grad-CAM visualizations to expose common failure modes and detector biases. TalkingHeadBench is hosted on https://huggingface.co/datasets/luchaoqi/TalkingHeadBench with open access to all data splits and protocols. Our benchmark aims to accelerate research towards more robust and generalizable detection models in the face of rapidly evolving generative techniques.
BibTeX
@inproceedings{xiong2026talkingheadbench,
  author = {Xinqi Xiong and Prakrut Patel and Qingyuan Fan and Amisha Wadhwa and Sarathy Selvam and Xiao Guo and Luchao Qi and Xiaoming Liu and Roni Sengupta},
  title = {{TalkingHeadBench: A Multi-Modal Benchmark \& Analysis of Talking-Head DeepFake Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
}
Figure