Syn-Att: Synthetic Speech Attribution via Semi-Supervised Unknown Multi-Class Ensemble of CNNs

Md Awsafur Rahman, Bishmoy Paul, Najibul Haque Sarker, Zaber Ibn Abdul Hakim, Shaikh Anowarul Fattah, Mohammad SaquibarXiv 2023 · DOI · arXiv

Abstract

The proliferation of high-fidelity synthetic speech necessitates robust forensic tools for both detection and attribution. This paper introduces Syn-Att, a framework designed to attribute synthetic audio to its generating algorithm. By employing a multi-class ensemble of Convolutional Neural Networks (CNNs) and a novel semi-supervised strategy for handling "Unknown" generation sources, Syn-Att addresses the open-set problem in audio forensics. Experimental results demonstrate that Syn-Att significantly outperforms state-of-the-art competitors, particularly under heavy audio distortion.

Introduction

While binary detection (real vs. fake) is a common focus in audio forensics, attribution—identifying the specific source algorithm—is crucial for tracking malicious content. However, traditional classifiers often fail when encountering "open-set" scenarios where the test audio comes from an unseen generator. Syn-Att aims to bridge this gap by creating a system capable of identifying five known algorithms while accurately categorizing unseen sources into an "Unknown" class.

Methodology

The Syn-Att pipeline is a multi-stage process that transforms raw audio into actionable forensic intelligence.

Overview of the Syn-Att pipeline, including feature extraction, ensemble modeling, and semi-supervised pseudo-labeling.

Feature Extraction and Data Augmentation

Raw audio is converted into log-mel spectrograms. To handle varying noise levels, two configurations were used: a standard resolution for clean audio and a higher-resolution version for noisy environments. Data augmentation techniques like MixUp, CutMix, and SpecAugment were applied to ensure the model focuses on algorithmic artifacts rather than speaker-specific characteristics.

The "Unknown" Class Strategy

To mitigate "hallucination" where the model incorrectly assigns unseen algorithms to known classes, the authors explicitly trained an "Unknown" category. This class was populated using a diverse mix of natural speech (from LJSpeech, LibriSpeech, and VCTK) and synthetic audio from external text-to-speech models.

Ensemble of CNNs and Semi-Supervised Learning

Syn-Att utilizes an ensemble of diverse CNN backbones, including ResNet, EfficientNet, and ConvNeXt. The final prediction is an average of the probability outputs, enhancing resilience against perturbations. Furthermore, a semi-supervised approach using pseudo-labeling allows the model to adapt to the specific distribution of the test data without requiring ground-truth labels.

Experimental Results

The framework was evaluated against the IEEE SP Cup 2022 benchmarks, which include weakly perturbed (Eval 1) and strongly perturbed (Eval 2) datasets.

Comparison of the proposed Syn-Att method against existing forensic techniques across different evaluation parts.

As shown in the comparative analysis, Syn-Att achieves a significant performance lead. On the challenging Eval 2 dataset, it outperformed the next best competitors by 12-13% in accuracy. On the cleaner Eval 1 dataset, it maintained a high accuracy of 98%. The results highlight the effectiveness of the ensemble approach and the "Unknown" class strategy in handling complex, real-world audio distortions.

Conclusion

Syn-Att provides a robust solution for the forensic attribution of synthetic speech. By combining diverse CNN architectures with a strategic approach to open-set classification and semi-supervised adaptation, it sets a new benchmark for accuracy and reliability in the field. Future work will explore reducing computational overhead and integrating transformer-based architectures for further performance gains.

Summary updated

Abstract

With the huge technological advances introduced by deep learning in audio & speech processing, many novel synthetic speech techniques achieved incredible realistic results. As these methods generate realistic fake human voices, they can be used in malicious acts such as people imitation, fake news, spreading, spoofing, media manipulations, etc. Hence, the ability to detect synthetic or natural speech has become an urgent necessity. Moreover, being able to tell which algorithm has been used to generate a synthetic speech track can be of preeminent importance to track down the culprit. In this paper, a novel strategy is proposed to attribute a synthetic speech track to the generator that is used to synthesize it. The proposed detector transforms the audio into log-mel spectrogram, extracts features using CNN, and classifies it between five known and unknown algorithms, utilizing semi-supervision and ensemble to improve its robustness and generalizability significantly. The proposed detector is validated on two evaluation datasets consisting of a total of 18,000 weakly perturbed (Eval 1) & 10,000 strongly perturbed (Eval 2) synthetic speeches. The proposed method outperforms other top teams in accuracy by 12-13% on Eval 2 and 1-2% on Eval 1, in the IEEE SP Cup challenge at ICASSP 2022.
BibTeX
@misc{rahman2023syn,
  author = {Md Awsafur Rahman and Bishmoy Paul and Najibul Haque Sarker and Zaber Ibn Abdul Hakim and Shaikh Anowarul Fattah and Mohammad Saquib},
  title = {{Syn-Att: Synthetic Speech Attribution via Semi-Supervised Unknown Multi-Class Ensemble of CNNs}},
  year = {2023},
  eprint = {2309.08146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.08146},
}
Figure