TL;DR
As voice cloning technologies evolve at a breakneck pace, audio deepfake detectors frequently fail when encountering speech generated by new, unseen text-to-speech (TTS) models. To solve this, ADD-GP combines a self-supervised speech foundation model (XLS-R) with a Gaussian Process classifier via Deep Kernel Learning. This framework allows the detector to adapt to entirely new deepfake generators using only a handful of examples (few-shot) or even a single example from a specific speaker (one-shot), all without suffering from catastrophic forgetting.
What Problem Does This Paper Address?
Current Audio Deepfake Detection (ADD) systems are highly effective at spotting synthetic speech from generation models they were trained on. However, they generalize poorly to unseen, state-of-the-art TTS engines. Because new voice cloning tools are released constantly—and many are locked behind paid APIs that prevent large-scale data collection—re-training massive neural networks from scratch is impractical.
Furthermore, when traditional deep learning models are fine-tuned on a few samples of a new deepfake style, they often suffer from catastrophic forgetting, meaning they lose their ability to detect older, previously learned deepfakes. The field lacks a method that can quickly, efficiently, and safely adapt to new generative threats with minimal data.
Methods
The authors introduce ADD-GP, a pipeline-based framework that merges deep representation learning with the mathematical flexibility of Gaussian Processes (GPs).
- Front-End Feature Extraction: The system uses XLS-R, a massive self-supervised speech foundation model based on Wav2Vec2, to extract robust audio embeddings. During the initial training phase, only the very last block of XLS-R is updated.
- Back-End Gaussian Process Classifier: Instead of a standard softmax layer, the backend is a Dirichlet-based Gaussian Process classifier.
- Deep Kernel Learning (DKL): DKL bridges the two components by applying a classical Radial Basis Function (RBF) kernel directly to the deep embeddings produced by the neural network.
- Few-Shot Adaptation: Because GPs are non-parametric, adapting the model to a new TTS engine does not require gradient descent or fine-tuning the weights. Instead, the few-shot examples from the new engine are simply added to a reference set of known samples, and the prediction is recomputed via a closed-form mathematical solution.
- MixPro Augmentation: To boost performance in data-starved scenarios, the authors incorporate MixPro, a data augmentation technique that blends the embeddings of known deepfakes with the new target deepfakes.
Datasets
To rigorously evaluate few-shot adaptation, the authors introduced a new benchmark dataset:
- LibriFake: Derived from the popular LibriSpeech corpus, the authors replicated the dataset using five state-of-the-art voice cloning and TTS models: yourTTS, Whisper-Speech, Vall-e-x, F5-TTS, and ElevenLabs.
- ElevenLabs as the OOD Target: ElevenLabs was intentionally withheld during initial training to act as the challenging, out-of-distribution (OOD) unseen model for few-shot testing.
- VoxCeleb: A subset of 30 speakers from the VoxCeleb dataset was used to evaluate the framework's performance in personalized, speaker-specific deepfake detection.
Key Insights and Ideas
- The Non-Parametric Advantage: Because Gaussian Processes rely on measuring similarity between data points rather than updating static weights, adding new examples to the model's "memory" naturally expands its capabilities without overwriting what it previously learned.
- Personalization Lowers the Barrier: If you know exactly whose voice you are trying to protect (e.g., a corporate executive or a high-profile individual), you can build a personalized GP detector. The authors discovered that a personalized detector requires significantly less data, achieving remarkable accuracy with just a single deepfake sample from that speaker.
- Uncertainty Quantification: Unlike standard deep learning classifiers that output uncalibrated scores, the Bayesian nature of ADD-GP provides well-calibrated probabilities. This means the system can reliably communicate how confident it is in its decision, which is crucial for real-world deployment.
Key Results
When tested against an entirely unseen deepfake generator (ElevenLabs), standard baseline models completely failed, yielding Equal Error Rates (EER) between 35% and 42% (near random guessing).
- Few-Shot Performance: With just 5 samples of ElevenLabs audio and MixPro augmentation, ADD-GP slashed the EER to 7.89%. With 100 samples, the EER dropped to an impressive 0.54%.
- Zero Catastrophic Forgetting: While baseline models saw their performance on original training data degrade after fine-tuning, ADD-GP maintained a near-perfect 0.1% EER on in-distribution data.
- Personalized One-Shot Detection: In speaker-specific scenarios, ADD-GP achieved a stunning 0.61% EER on LibriFake using just a single deepfake sample (1-shot) for adaptation.
Limitations and Future Work
While the personalized deepfake detection results are incredibly powerful, the authors note a practical limitation: it requires deepfake samples specifically generated for the target speaker to adapt effectively, rather than relying on generic deepfake samples from the new TTS model.
Additionally, though not heavily detailed in the text, scaling Gaussian Processes to massive datasets traditionally introduces computational bottlenecks. The authors sidestepped this by using a fixed reference set of 1,000 samples, but future work could explore how the system scales as the library of known generative threats grows into the tens of thousands.
Summary updated