MOS-FAD: Improving Fake Audio Detection Via Automatic Mean Opinion Score Prediction

Wangjin Zhou, Zhengdong Yang, Chenhui Chu, Sheng Li, Raj Dabre, Yi Zhao, Tatsuya KawaharaarXiv 2024 · arXiv

TL;DR

MOS-FAD introduces a novel framework that repurposes automatic Mean Opinion Score (MOS) prediction—traditionally used to evaluate synthetic speech quality—to dramatically improve Fake Audio Detection (FAD). By leveraging MOS for both targeted training data selection and as a gating mechanism in model fusion, the system achieves a 13.6% relative reduction in Equal Error Rate (EER) over the previous state-of-the-art on challenging deepfake benchmarks.

What Problem Does This Paper Address?

Generative speech technologies like Text-to-Speech (TTS) and Voice Conversion (VC) have advanced to the point where synthetic voices sound remarkably human. While beneficial for many applications, these technologies also enable sophisticated spoofing attacks and deepfakes.

Existing Fake Audio Detection (FAD) systems typically treat this as a binary classification problem, relying on acoustic front-ends to distinguish real from fake. However, they face two major hurdles:

  1. Data Imbalance: Standard FAD training datasets are heavily skewed, often containing up to ten times more fake samples than real ones.
  2. The "Hard Case" Problem: As synthetic speech gets better, the boundary between real and fake blurs. Standard models waste capacity learning from obvious, low-quality fakes rather than focusing on the highly realistic fakes that pose the actual security threat.

Methods

The authors introduce MOS-FAD, a framework that injects external knowledge from an automatic MOS predictor into the FAD pipeline at two critical stages: data filtering and model fusion.

Fig. 1. Proposed model structure.

1. The Core Architecture

The framework utilizes 7 distinct self-supervised learning (SSL) speech models (including variants of Wav2Vec 2.0, HuBERT, and WavLM). These models are duplicated into two parallel subsystems:

  • SSL-FADs: Fine-tuned to output binary FAD scores ($y_i$).
  • SSL-MOSs: Fine-tuned to predict continuous speech quality scores ($z_i$) on a 1-to-5 scale, which are then combined via an MOS Fusion layer into a final quality score ($z_f$).

2. MOS-Based Data Filtering

Instead of training on the entire, imbalanced dataset, the authors use the predicted MOS scores to filter the training data. They select only the samples falling into the "uncertainty zone" (MOS between 3.0 and 4.0), effectively balancing the dataset and forcing the FAD models to train exclusively on highly realistic fakes and natural human speech.

3. Advanced Model Fusion

To combine the predictions of the 7 FAD models and the MOS predictor, the authors explored multiple fusion strategies:

  • Gated MLP: Uses a linear decoder and a Sigmoid function on the MOS score to act as a dynamic "gate," controlling how much information from the FAD scores passes into the final Multi-Layer Perceptron.
  • Fusion with Thresholding: A hybrid approach where samples with extremely low MOS scores ($<2.5$) are automatically flagged as fake (0), and those with extremely high scores ($>4.0$) are flagged as real (1). The Gated MLP is only invoked for the ambiguous middle ground.

Datasets

  • VoiceMOS Challenge 2022 (Main Track): Used exclusively to train the automatic MOS predictor.
  • ASVspoof2019 LA Track: Used for training and validating the FAD models, as well as evaluating the MOS-based data filter.
  • ASVspoof2021 DF Track (Evaluation Set): A massive test set comprising 611,829 samples processed with various commercial audio codecs, used to evaluate the robustness of the final system against real-world compression and coding artifacts.

Key Insights and Ideas

  • The Uncertainty Zone: By plotting the MOS distributions of real vs. fake audio, the authors discovered that fake audio rarely scores above 4.0, and real audio rarely scores below 2.5. The overlap between 3.0 and 4.0 represents the true battleground for FAD systems.
  • Quality as a Proxy for Authenticity: A binary label (real/fake) throws away granular information. A continuous MOS score provides a fine-grained metric of "naturalness," which serves as an excellent prior for determining if an audio clip is a high-quality spoof.
  • MOS as a Gatekeeper: In model fusion, treating the MOS score as a gating mechanism (rather than just another feature input) yielded significantly better results, proving that quality metrics can effectively modulate how acoustic classification features are interpreted.
Fig. 3. The MOS score distributions in ASVspoof2019 LA track.

Key Results

  • Data Filtering Breakthrough: Training the base SSL models on the MOS-filtered dataset yielded massive performance gains across the board. For example, the Equal Error Rate (EER) for the standard Wav2Vec Large model plummeted from 0.4794 (without the filter) to 0.1891 (with the filter).
  • New State-of-the-Art: The proposed Gated MLP + thresholding fusion method achieved an EER of 0.1351 on the ASVspoof2021 DF evaluation set. This comfortably beat the previous SOTA benchmark of 0.1564, representing a 13.6% relative reduction in error.

Limitations and Future Work

The authors do not explicitly dedicate a section to limitations, but a few factors are apparent from the text:

  • Dependency on MOS Accuracy: The framework's success heavily relies on the robustness of the external automatic MOS predictor. If the MOS predictor fails to generalize to a new type of synthetic speech, the data filter and thresholding mechanisms could miscategorize samples.
  • Future Directions: The authors note that because the "fusion with thresholding" approach is model-agnostic, it can be applied to any existing FAD architecture. Future work will likely explore integrating this framework with alternative, non-SSL-based front-ends and testing it against emerging, even more advanced generative speech algorithms.

Summary updated

Abstract

Automatic Mean Opinion Score (MOS) prediction is employed to evaluate the quality of synthetic speech. This study extends the application of predicted MOS to the task of Fake Audio Detection (FAD), as we expect that MOS can be used to assess how close synthesized speech is to the natural human voice. We propose MOS-FAD, where MOS can be leveraged at two key points in FAD: training data selection and model fusion. In training data selection, we demonstrate that MOS enables effective filtering of samples from unbalanced datasets. In the model fusion, our results demonstrate that incorporating MOS as a gating mechanism in FAD model fusion enhances overall performance.
BibTeX
@misc{zhou2024mos,
  author = {Wangjin Zhou and Zhengdong Yang and Chenhui Chu and Sheng Li and Raj Dabre and Yi Zhao and Tatsuya Kawahara},
  title = {{MOS-FAD: Improving Fake Audio Detection Via Automatic Mean Opinion Score Prediction}},
  year = {2024},
  eprint = {2401.13249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.13249},
}
Figure