Learning Efficient Representations for Fake Speech Detection

Nishant Subramani, Delip RaoAAAI 2020 · DOI

Introduction

As speech synthesis (SS) and voice conversion (VC) technologies like WaveNet and Tacotron advance, "deepfake" audio has become increasingly indistinguishable from human speech. This evolution poses significant security risks, including misinformation, identity theft, and financial fraud. A primary challenge in synthetic speech detection is the "shifting adversary"—new synthesis models are released constantly, and traditional detectors often fail when encountering an "attack vector" (synthesis method) unseen during training. Furthermore, for practical deployment, detectors must be computationally efficient enough to run on mobile edge devices.

This paper introduces EfficientCNN and RES-EfficientCNN, two ultra-lightweight deep learning models designed to address these challenges. By leveraging multi-task learning and transfer learning, these models achieve state-of-the-art accuracy with a minimal memory footprint, enabling rapid adaptation to new voice-cloning attacks.

Methodology

The authors propose two primary architectures based on a variant of LightCNN:

  1. EfficientCNN: Comprises an input processing block, four convolutional blocks, and a classification head.
  2. RES-EfficientCNN: Incorporates residual (skip) connections within the convolutional blocks to capture more complex features and enhance training stability.

Multi-Task Learning and Transfer Learning

A key innovation is the use of multi-task learning. Beyond the binary classification of "real vs. fake," the models are trained to identify the specific system that generated the spoofed audio (e.g., a specific neural network or unit-selection system). This auxiliary task forces the model to learn richer representations of synthetic artifacts, which significantly boosts primary detection performance.

The multi-task learning architecture employs dual classification heads to identify both the authenticity and the specific source of the audio.

To handle emerging threats, the researchers utilize transfer learning. By fine-tuning a pre-trained model on a small subset of data from a new attack vector, the system can adapt to novel synthesis methods with extreme sample efficiency.

Datasets

The models were evaluated on two primary datasets:

  • ASVSpoof2019 (Logical Access): A standard benchmark featuring bonafide speech and spoofs from six different SS and VC systems.
  • RTVCSpoof: A new dataset introduced by the authors using the Real-Time Voice Cloning (RTVC) toolkit to simulate modern, "in-the-wild" adversaries. It includes over 8,000 utterances from 369 speakers with diverse backgrounds.

Results and Discussion

The experimental results highlight several critical advantages of the proposed approach:

  • Implicit Supervision via Multi-Tasking: Adding the source detection task improved F1 scores by up to 11% in data-sparse scenarios. Crucially, this improvement comes with a negligible 0.5% increase in parameter count, and source labels are not required during inference.
  • Extreme Sample Efficiency: In adversarial settings, transfer learning matched or exceeded fully supervised performance using as little as 6.25% of the available training data (approximately 240 examples).
Comparison of Macro F1 scores between single-task and multi-task learning across varying training data sizes, demonstrating the superiority of multi-tasking in low-data regimes.
  • Efficiency and Accuracy: The models feature fewer than 50,000 parameters and a memory footprint under 100 KB. The largest variant uses only 256 MFLOPs—roughly 100x less compute than a standard ResNet-18. Despite this small size, Multi-RES-EfficientCNN achieved a 97.61 macro-F1 score on ASVSpoof2019 and a 99.12 F1 score on RTVCSpoof via transfer learning.

Conclusion and Future Work

While highly effective for clean digital audio, the current study focuses on "Logical Access" and does not yet account for "Physical Access" challenges like background noise or microphone distortion. Future work will explore alternative input representations, such as raw-waveform or phase-based features, to better detect subtle synthesis artifacts. Additionally, improving generalization to entirely different classes of synthesis without fine-tuning remains an open area for research.

Summary updated

BibTeX
@inproceedings{subramani2020learning,
  author = {Nishant Subramani and Delip Rao},
  title = {{Learning Efficient Representations for Fake Speech Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.6044},
}
Figure