The DeepSpeak Dataset

Sarah Barrington, Maty Bohacek, Hany FaridCVPR 2026 Findings 2026 · OpenReview

TL;DR

DeepSpeak is a newly introduced, ethically collected dataset containing over 100 hours of high-quality, multimodal "talking head" video and audio designed to train and evaluate deepfake detectors. Large-scale benchmarking reveals a severe generalization gap: current state-of-the-art deepfake detectors fail dramatically when evaluated on these modern, generative AI-produced fakes without explicit retraining.

What Problem Does This Paper Address?

As generative AI tools rapidly evolve, deepfakes have transitioned from low-quality novelties to hyper-realistic threats capable of fueling disinformation, financial fraud, and video-conference impersonation (such as impostor hiring). To counter this, the digital forensics community relies heavily on benchmark datasets to train detection algorithms.

However, existing datasets suffer from critical flaws:

  • Outdated Technology: They rely on obsolete deepfake generators that do not reflect the current state of the art.
  • Ethical Issues: Many datasets consist of unconsented data scraped directly from public online repositories.
  • Lack of Realism: They rarely employ identity-matching protocols, resulting in face-swaps between completely dissimilar people—an artifact that real-world attackers easily avoid by choosing lookalike targets.

DeepSpeak addresses these gaps by providing a diverse, ethically sourced, and highly realistic dataset specifically focused on the challenging "talking heads" video-conferencing context.

Methods

The creation of DeepSpeak follows a rigorous four-step pipeline:

Figure 1. An overview of the DeepSpeak Dataset sourced from a diverse selection of consenting participants using a custom-built data collection methodology. The dataset also comprises deepfakes generated from 14 video and three audio deepfake methods using facial identity matching to improve the realism of the generated deepfakes.
  1. Data Collection: The authors built an open-source, web-based application to collect phonetically rich scripted and unscripted video recordings from consenting participants.
  2. Perceptual Identity Matching: To simulate sophisticated, targeted real-world attacks, the authors paired participants based on facial similarity. Instead of traditional biometric matching (like ArcFace), they extracted average CLIP embeddings from video frames and applied an agglomerative clustering algorithm. This approach yielded qualitatively superior matching, particularly for women and people of color.
  3. Multimodal Generation: Using the matched pairs, the authors deployed 14 video synthesis engines (spanning face-swap, lip-sync, and avatar generators) and 3 commercial voice-cloning services (ElevenLabs, PlayAI, Speechify) to create highly convincing deepfakes.
  4. Automated Validation: An automated input/output filtering suite was developed to detect and remove common deepfake generation failures, such as malformed facial structures, frozen expressions, or corrupted frames.

Datasets

The paper introduces DeepSpeak (Ours, 2025), which consists of:

  • 50+ hours of authentic footage from 500 diverse, paid, and consenting crowdworkers.
  • 50+ hours of deepfake footage (14,005 videos total across face-swap, lip-sync, avatar, and cloned audio modalities).

The dataset is split into an 80/20 training/testing ratio with zero overlap in facial or vocal identities. The authors compared DeepSpeak against 14 historical datasets released over the last seven years (including DFDC, Celeb-DF, and FakeAVCeleb), highlighting DeepSpeak's unique combination of participant consent, identity matching, and modern multimodal generation.

Key Insights and Ideas

  • The Generalization Failure: Deepfake detectors are highly brittle. Models trained on older datasets completely fail to recognize fakes generated by modern tools, indicating that detectors are learning dataset-specific artifacts rather than generalized properties of synthetic media.
  • CLIP for Biometric Matching: A surprising finding was that CLIP embeddings outperformed specialized face-recognition models like ArcFace for perceptual identity matching. CLIP-based matching improved Frechet Inception Distance (FID) by 4% and Learned Perceptual Image Patch Similarity (LPIPS) by 6%, proving highly effective at clustering lookalikes across diverse demographics.
  • Severe Class Bias: Out-of-the-box detectors exhibited massive class biases on the new dataset—for instance, one top video detector achieved 98.8% accuracy on real videos but caught only 3.5% of the deepfakes.

Key Results

The authors benchmarked several state-of-the-art video detectors (FreqNet, GenConViT, LipFD) and audio detectors (AASIST, RawNet2, RawGAT-ST, and foundation model embeddings):

  • Video Detection: Pretrained models failed to generalize to DeepSpeak. For example, FreqNet's F1-score dropped to a dismal 0.2 when evaluated on DeepSpeak out-of-the-box. Fine-tuning on DeepSpeak restored high performance (F1 > 0.9), but caused a catastrophic drop in performance on the models' original training datasets.
  • Audio Detection: Raw waveform models like AASIST performed exceptionally well when trained directly on DeepSpeak (98.8% accuracy). However, pretrained versions of AASIST dropped to roughly 60% accuracy when tested on DeepSpeak out-of-the-box.
  • Combined Audio-Visual Detection: Testing a joint detection method based on audio-to-lip sync mismatch yielded an accuracy of only 65.5% on real videos and 67.9% on fakes, proving that multimodal detectors face the same generalization hurdles.

Limitations and Future Work

  • Language Constraints: Due to a lack of high-quality, open-source deepfake engines for non-English languages, DeepSpeak currently only features English-speaking participants.
  • Open-Source Video Bias: All video generators used in this study are open-source; commercial video generation models are not yet represented.
  • Video-Level Manipulation: The dataset currently treats videos as binary (entirely real or entirely fake). It does not yet include targeted, word-level edits within a real video.

To keep pace with the rapid evolution of generative AI, the authors plan to release updated partitions of the dataset (v1, v2, etc.) one to two times per year.

Summary updated

Abstract

Deepfakes represent a growing concern across domains such as disinformation, fraud, and non-consensual media. In particular, the rise of video conference and identity-driven attacks in high-stakes scenarios--such as impostor hiring--demands new forensic resources. Despite significant efforts to develop robust detection classifiers to distinguish the real from the fake, commonly used training datasets remain inadequate: relying on low-quality and outdated deepfake generators, consisting of content scraped from online repositories without participant consent, lacking in multimodal coverage, and rarely employing identity-matching protocols to ensure realistic fakes. To overcome these limitations, we present the DeepSpeak dataset, a diverse and multimodal dataset comprising over 100 hours of authentic and deepfake audiovisual content, specifically focused on the challenging and diverse "talking heads" context. We contribute: i) more than 50 hours of real, self-recorded data collected from 500 diverse and consenting participants, ii) more than 50 hours of state-of-the-art audio and visual deepfakes generated using 14 video synthesis engines and three voice cloning engines, and iii) an embedding-based, identity-matching approach to ensure the creation of convincing, high-quality identity face swaps that realistically simulate adversarial deepfake attacks. We also perform large-scale evaluations of state-of-the-art deepfake detectors and show that, without retraining, these detectors fail to generalize to this DeepSpeak dataset, highlighting the importance of a large and diverse dataset containing deepfakes from the latest generative-AI tools. https://github.com/hfaridlab/deepspeak
BibTeX
@inproceedings{barrington2026deepspeak,
  author = {Sarah Barrington and Maty Bohacek and Hany Farid},
  title = {{The DeepSpeak Dataset}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=OeU1j4UbZa},
}

Loading related content...

Figure