Uncertainty as a Predictor: Leveraging Self-Supervised Learning for Zero-Shot MOS Prediction

Aditya Ravuri, Erica Cooper, Junichi YamagishiWorkshops @ ICASSP 2024 · DOI

TL;DR

This paper explores using the inherent uncertainty of self-supervised learning (SSL) models, like wav2vec, as a proxy for predicting Mean Opinion Scores (MOS) in audio quality assessment. They demonstrate that these uncertainty measures, derived directly from pre-trained models without task-specific fine-tuning, correlate surprisingly well with human judgments of audio quality, offering a promising approach for low-resource scenarios.

What Problem Does This Paper Address?

Predicting audio quality, often measured by Mean Opinion Scores (MOS), is crucial for evaluating speech synthesis and voice conversion systems. However, collecting MOS data through large-scale listening tests is labor-intensive and time-consuming. This is particularly problematic in low-resource settings where obtaining sufficient labeled data for traditional supervised models is infeasible. The paper aims to bridge this gap by investigating whether readily available, pre-trained SSL models can predict MOS without requiring any MOS-specific training data, a scenario known as zero-shot prediction.

Methods

The core idea is to leverage the "uncertainty" of pre-trained SSL models as a proxy for audio quality. The authors propose several "Uncertainty Measures" (UMs) derived from the output logits of SSL models:

  • Entropy: Calculated from the probability distribution of predicted tokens/latent codes for each time window, then averaged. High entropy suggests the model is uncertain about what is being uttered.
  • Logits' Mean, Max, and Standard Deviation: These are computationally cheaper alternatives to entropy. A high maximum logit suggests high confidence in a particular token, potentially correlating with better quality. A high standard deviation might indicate uncertainty.

They hypothesize that high model uncertainty around the content of an audio sequence corresponds to low audio quality. The paper also explores "handicapping" wav2vec2-based ASR models by introducing dropout and averaging multiple forward passes to see if this artificially induced uncertainty improves MOS prediction.

Datasets

The primary datasets used are from the VoiceMOS 2022 and 2023 challenges. These challenges provide audio files from synthesized speech and voice conversion systems, paired with MOS ratings obtained from listening tests. The paper considers several tracks:

  • Spoken English (main track of VoiceMOS 2022)
  • Spoken Chinese (out-of-domain track of VoiceMOS 2022 and Track 3 of VoiceMOS 2023)
  • Spoken French (Track 1 of VoiceMOS 2023)
  • English Singing Audio Quality Evaluation (Track 2 of VoiceMOS 2023)

The experiments are conducted in a zero-shot setting, meaning no MOS scores are used for training the SSL models.

Key Insights and Ideas

  • Uncertainty as a Quality Proxy: The most significant insight is that the inherent uncertainty of out-of-the-box, pre-trained SSL models (like wav2vec) can serve as a surprisingly effective predictor of audio quality (MOS). This bypasses the need for extensive MOS data.
  • wav2vec Outperforms wav2vec2: Counter-intuitively, the older wav2vec models, when used for uncertainty estimation, showed better MOS prediction performance than the more advanced wav2vec2 models. The authors hypothesize this might be due to wav2vec's contrastive predictive probabilities being more directly related to audio quality and potentially better model calibration.
  • Handicapping Improves wav2vec2: Artificially introducing uncertainty into wav2vec2-based ASR models (via dropout) actually improves their MOS prediction performance, supporting the core hypothesis.
  • Connection to Intelligibility: The uncertainty measures derived from SSL models show a correlation with intelligibility measures (like Word Error Rate), which in turn correlate with MOS scores, suggesting a shared underlying factor related to audio signal clarity.

Key Results

  • For the spoken English track, wav2vec models achieved Spearman correlation coefficients (SRCC) of around 70% with MOS scores using their uncertainty measures, without any task-specific training.
  • Handicapping wav2vec2-based ASR models with dropout increased their SRCC by up to 10%.
  • The uncertainty measures from wav2vec2-based ASR models correlated with intelligibility measures (WER) at about 70%, while intelligibility measures correlated with MOS at about -53%.
  • Performance on non-English datasets was generally poorer but still showed weak positive correlations that improved with model handicapping.
  • Further pre-training wav2vec on target audio data (without MOS scores) boosted performance to above 70% SRCC on several non-English tasks.

Limitations and Future Work

  • Performance Gap: While the SSL uncertainty measures are strong predictors, they do not yet reach state-of-the-art performance compared to models specifically trained on MOS data (e.g., the baseline in [1] achieved 92% SRCC on spoken English).
  • Model Calibration: The authors suggest that wav2vec2 models might be "too good" and less calibrated for this specific task, making it harder for their uncertainties to reflect subtle quality differences.
  • Non-English Tasks: Performance on non-English tasks was weaker, though improved with domain-adaptive pre-training.
  • Future Work: The authors suggest extending this approach to other domains where understanding out-of-distribution data is critical, such as dimensionality reduction. They also propose further exploration into the differences between UM types and their performance across various contexts. Investigating the impact of different SSL architectures and pre-training strategies on uncertainty estimation for MOS prediction is also a natural next step.

Summary updated

BibTeX
@inproceedings{ravuri2024uncertainty,
  author = {Aditya Ravuri and Erica Cooper and Junichi Yamagishi},
  title = {{Uncertainty as a Predictor: Leveraging Self-Supervised Learning for Zero-Shot MOS Prediction}},
  booktitle = {Workshops @ ICASSP},
  year = {2024},
  doi = {10.1109/icasspw62465.2024.10626267},
}
Figure