Training-Free Deepfake Voice Recognition by Leveraging Large-Scale Pre-Trained Models

Alessandro Pianese, Davide Cozzolino, Giovanni Poggi, Luisa Verdoliva

Introduction

The rapid advancement of generative artificial intelligence has led to the creation of highly realistic synthetic voices, posing significant security and ethical challenges. Traditional deepfake detection methods typically rely on supervised learning, where models are trained on large datasets of both real and fake audio. However, these models often struggle with "out-of-distribution" data—new synthesis techniques they have not encountered during training. This paper presents a paradigm shift: a training-free detection framework that treats deepfake recognition as a speaker verification problem rather than a binary classification task.

The Person-of-Interest (PoI) Framework

Instead of attempting to model the infinite variety of synthetic artifacts, the proposed method focuses on the identity of the speaker. This Person-of-Interest (PoI) approach assumes that for any suspicious audio clip, a small set of "pristine" (guaranteed real) reference recordings of the alleged speaker is available.

The detection pipeline consists of four main stages:

  1. Reference Acquisition: A small collection of verified audio segments from the target speaker is gathered.
  2. Feature Extraction: Both the reference clips and the test audio are processed through a large-scale pre-trained foundation model. These models are used "off-the-shelf" with frozen weights; no additional training or fine-tuning on deepfake data is required.
  3. Similarity Scoring: The system generates mathematical embeddings for each clip and calculates the cosine similarity between the test embedding and each reference embedding.
  4. Decision Logic: The final score is determined by the maximum similarity value. If the test clip does not sufficiently match any of the reference clips, it is classified as a deepfake.

Experimental Setup and Foundation Models

The researchers evaluated several state-of-the-art foundation models to determine which features are most robust for identity verification: Wav2Vec2-xlsr, AudioCLIP, LaionCLAP, and BEATs (Bidirectional Emotional Analytics and Transcription).

Testing was conducted across three diverse benchmarks to measure generalization:

  • ASVSpoof2019: A standard laboratory dataset featuring 13 different synthesis and conversion methods.
  • ASVSpoof2021: A dataset incorporating simulated telephony effects and compression to mimic real-world transmission.
  • InTheWild: A dataset comprising real-world audio from 54 different identities collected from public sources, representing a truly "unseen" scenario for models trained in controlled environments.

Key Findings: The Superiority of Semantic Features

The most significant finding was the exceptional performance of the BEATs model. The researchers discovered that BEATs significantly outperforms other models because it utilizes a "semantic tokenizer." While many audio models focus on low-level signal reconstruction, BEATs prioritizes high-level meaning and identity-related features. This makes it remarkably difficult for a deepfake to bypass the system by simply mimicking surface-level acoustic characteristics.

In the "InTheWild" evaluation, supervised models like AASIST saw their performance collapse to an Area Under the Curve (AUC) of 60.12%, barely outperforming a random guess. In contrast, the training-free BEATs-based method maintained a near-perfect AUC of 99.73%. Furthermore, the system proved highly efficient, requiring as few as five reference clips to achieve an AUC greater than 0.9.

Limitations and Conclusion

While the training-free approach offers a robust solution to the generalization problem, it is currently limited to scenarios where the speaker's identity is known and reference audio is available. It cannot detect a deepfake of an anonymous or unknown individual. Additionally, the system's accuracy is sensitive to the quality and diversity of the reference set; a significant environmental mismatch between reference and test audio could potentially trigger false positives.

By bypassing the need for synthetic training data, this framework offers a sustainable path forward in the "cat-and-mouse" game of deepfake detection. The success of semantic-rich models like BEATs suggests that focusing on high-level identity features is a more effective strategy for forensic audio analysis than chasing the ever-evolving artifacts of generative AI.

Summary updated

BibTeX
@inproceedings{pianese2024training,
  author = {Alessandro Pianese and Davide Cozzolino and Giovanni Poggi and Luisa Verdoliva},
  title = {{Training-Free Deepfake Voice Recognition by Leveraging Large-Scale Pre-Trained Models}},
  booktitle = {IH\&MMSec},
  year = {2024},
  url = {https://openreview.net/forum?id=I2uwOeqJLq},
}
Figure