Abstract: Significant advancements made in the generation of deepfakes have caused security and privacy issues. Attackers can easily impersonate a person's identity in an image by replacing his face with the target person's face. Moreover, a new domain of cloning human voices using deep-learning technologies is also emerging. Now, an attacker can generate realistic cloned voices of humans using only a few seconds of audio of the target person. With the emerging threat of potential harm deepfakes can cause, researchers have proposed deepfake detection methods. However, they only focus on detecting a single modality, i.e., either video or audio. On the other hand, to develop a good deepfake detector that can cope with the recent advancements in deepfake generation, we need to have a detector that can detect deepfakes of multiple modalities, i.e., videos and audios. To build such a detector, we need a dataset that contains video and respective audio deepfakes. We were able to find a most recent deepfake dataset, Audio-Video Multimodal Deepfake Detection Dataset (FakeAVCeleb), that contains not only deepfake videos but synthesized fake audios as well. We used this multimodal deepfake dataset and performed detailed baseline experiments using state-of-the-art unimodal, ensemble-based, and multimodal detection methods to evaluate it. We conclude through detailed experimentation that unimodals, addressing only a single modality, video or audio, do not perform well compared to ensemble-based methods. Whereas purely multimodal-based baselines provide the worst performance.
Introduction
Most deepfake research to date has operated in silos. Video detectors typically look for visual artifacts like unnatural blinking or skin textures, while audio detectors focus on acoustic anomalies in synthesized speech. However, modern attackers are increasingly using tools like Wav2Lip to perfectly sync cloned voices with manipulated faces. Existing datasets, such as the popular Deepfake Detection Challenge (DFDC), often lack granular labels—they might mark a video as "fake" without specifying if the audio, the video, or both were altered. This makes it difficult to train and evaluate detectors that can handle "cross-domain" forgeries. This research seeks to establish a baseline for how well current technology can distinguish between real and fake when both modalities are in play.
Methodology
The study categorizes detection methods into three distinct approaches to evaluate the FakeAVCeleb dataset:
- Unimodal Detectors: Standard models trained on a single modality. For video, architectures like Xception, VGG16, and EfficientNet-B0 were used. For audio, raw sound was converted into Mel-frequency cepstral coefficients (MFCCs) to treat audio detection as an image classification problem.
- Ensemble Methods: A hybrid approach where separate audio and video models are trained independently, and their predictions are combined using Soft-voting (averaging probabilities) or Hard-voting (majority rule).
- Multimodal Detectors: Single models designed to process both audio and video features simultaneously. Because few "native" multimodal deepfake detectors exist, the authors adapted architectures originally designed for tasks like food recipe classification and movie genre prediction.
The FakeAVCeleb Dataset
The primary focus is the FakeAVCeleb dataset, built using 600 unique videos from the VoxCeleb2 dataset. It is racially and gender-balanced, featuring five ethnic groups. It contains three specific types of forgeries:
- Real video with fake (cloned) audio.
- Fake (faceswapped) video with real audio.
- Fake video with fake audio (lip-synced).
Generation tools included state-of-the-art software like Faceswap, FSGAN, and Real-Time Voice Cloning (RTVC).
Key Results and Insights
The experimental results revealed several critical findings regarding the current state of deepfake detection:
- The Multimodal Paradox: Purely multimodal models—those designed to fuse audio and video features into a single representation—performed the worst, often reaching only 50-67% accuracy. This suggests that deepfake detection requires looking for specific inconsistencies between modalities rather than just a general "fused" understanding.
- Ensembles are the "Least Bad" Option: Currently, running two specialized models in parallel (Ensemble) yields better results than trying to train one "jack-of-all-trades" model. The VGG16 ensemble achieved 78.04% accuracy.
- Unexpected Winners: VGG16, an older architecture, surprisingly outperformed the more modern Xception model on video detection in this specific dataset, achieving 81.03% accuracy.
- Audio is the Weak Link: Detection accuracy for audio-only fakes (maxing out at ~76.26% with Xception) was generally lower than for video, indicating that voice cloning technology is becoming exceptionally difficult to detect using standard image-based classifiers.
Conclusion and Future Work
The study acknowledges that the multimodal models tested were not originally designed for deepfake detection. These models were built to find similarities, whereas deepfake detection requires finding dissimilarities (e.g., mismatch between jaw movement and phonemes).
Future research should focus on developing "native" multimodal detectors specifically designed to spot synchronization errors between lip movements and speech. Additionally, investigating why certain architectures like Meso-4 fail dramatically on this dataset (43.15% accuracy) compared to others remains a priority for improving robustness.
Summary updated