Abstract
The task of deepfakes detection is far from being solved by speech or vision researchers. Several publicly available databases of fake synthetic video and speech were built to aid the development of detection methods. However, existing databases typically focus on visual or voice modalities and provide no proof that their deepfakes can in fact impersonate any real person. In this paper, we present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized and video have high visual and audio qualities. We demonstrate the vulnerability of state-of-the-art speaker and face recognition systems to these synthetic identities. Our assessment shows that by tuning existing pretrained deepfake models to specific identities, attackers can successfully spoof recognition systems in more than 90% of cases.
Introduction
While the internet is flooded with deepfake videos, most existing research databases focus on a single modality—either just video or just audio. Furthermore, many of these "fakes" are easily detectable because they lack high resolution, proper lip-sync, or realistic skin tones. A critical gap identified by researchers at the Idiap Research Institute is the lack of understanding regarding how well these deepfakes can impersonate a specific target. This research shifts the focus from simple detection to assessing the success of impersonation against biometric systems.
Methodology
To create realistic, synchronized audio-visual fakes, the researchers combined several state-of-the-art (SOTA) generative techniques:
- Video Generation: Using DeepFaceLab, the team employed DF and LIAE architectures at resolutions up to 320x320. A key finding was that "out-of-the-box" models are insufficient; fine-tuning models for 50,000 iterations for specific identity pairs is what makes a deepfake truly dangerous to biometric systems.
- Blending and Post-Processing: The "secret sauce" of a convincing deepfake lies in masking, color correction, and warping. Paradoxically, the post-processing used to make a fake look "real" to a human often masks the GAN-generated artifacts that detection algorithms rely on.
- Audio Generation: For the video dataset, Voice Conversion (VC) models like FreeVC and YourTTS were used to ensure synthetic voices remained perfectly synced with original lip movements.
Datasets: SWAN-DF and LibriTTS-DF
The authors introduced two new benchmarks to the research community:
- SWAN-DF: Based on the SWAN dataset of real HD videos, it contains over 150,000 video combinations across 60 pairs of identities. It features high visual and audio quality with synchronized lip movements.
- LibriTTS-DF: An audio-only dataset based on LibriTTS, utilizing the latest Text-to-Speech (TTS) methods like TorToiSe and Adaspeech to generate synthetic speech for 39 speakers.
Results and Vulnerability Assessment
The study utilized the Impostor Attack Presentation Match Rate (IAPMR) to measure the success of impersonation. A high IAPMR indicates that the recognition system frequently misidentified the fake as the target identity.
- Face Recognition: The MobileFaceNet system proved highly vulnerable. When using best-tuned 256px models, the IAPMR reached 97.36%, meaning the system almost always failed to distinguish the fake from the real target.
- Speaker Recognition: The ECAPA-TDNN system was also compromised. While zero-shot voice conversion was less effective, tuning the FreeVC model pushed the success rate to 94.21%.
- TTS Performance: In the LibriTTS-DF dataset, the diffusion-based TorToiSe TTS model achieved an IAPMR of 86.61% without identity-specific tuning, highlighting the growing threat of advanced audio synthesis.
Conclusion
The results demonstrate a significant vulnerability in current biometric security. The researchers conclude that environment and tuning are critical factors in deepfake effectiveness; for instance, synthetic speech from clean data (LibriTTS) was more effective than that from noisy environments (SWAN). Future research must prioritize multimodal detection—systems that analyze inconsistencies between audio and visual channels simultaneously—to counter these increasingly sophisticated identity-theft tools.
Summary updated