Quality Degradation Diagnosis for Voice Networks — Estimating the Perceived Noisiness, Coloration, and Discontinuity of Transmitted Speech

Gabriel Mittag, Sebastian MöllerInterspeech 2019

TL;DR

This paper introduces a novel deep learning model that goes beyond simply predicting overall speech quality. It can diagnose why speech quality is degraded by estimating three key perceptual dimensions: noisiness, coloration, and discontinuity. This allows for more targeted network optimization and a deeper understanding of audio quality issues.

What Problem Does This Paper Address?

Evaluating the quality of voice communication over networks is crucial for service providers. Traditionally, this involves subjective listening tests, which are time-consuming and expensive. Instrumental methods have been developed to automate this, but existing single-ended models (which don't require a clean reference signal) primarily predict an overall quality score (like MOS). While useful, these models don't tell us what is causing the degradation. Is it background noise, frequency distortions, or sudden interruptions? This lack of diagnostic information hinders effective troubleshooting and network improvement. This paper aims to fill this gap by developing a model that can not only predict overall quality but also pinpoint the specific perceptual dimensions responsible for any perceived degradation.

Methods

The proposed model builds upon the authors' previous work on a non-intrusive speech quality assessment model called NISQA. It employs a deep learning architecture consisting of two main components:

  1. Convolutional Neural Network (CNN): This part processes segments of the degraded speech signal (represented as log-Mel-spectrograms) to extract per-frame features. This is a standard approach for audio feature extraction.
  2. Recurrent Neural Network (RNN), specifically a Bidirectional Long Short-Term Memory (BiLSTM): The per-frame features from the CNN are fed into the BiLSTM, along with the per-frame quality estimates. The BiLSTM then aggregates these features over time to predict the three quality dimensions: noisiness, coloration, and discontinuity.

A key innovation is the decision to train three separate BiLSTM networks, one for each quality dimension, rather than a single network for all three. This allows for better balancing of training data for each dimension and leads to improved results. The CNN architecture is standard, employing convolutional layers, batch normalization, ReLU activations, and max-pooling, followed by fully connected layers. The BiLSTM architecture is also standard, using multiple layers with dropout for regularization.

Datasets

The model was trained and evaluated on a combination of existing and newly annotated datasets:

  • Existing Databases: Seven databases with subjective ratings for noisiness, coloration, and discontinuity were used.
  • Expert Annotated Databases: For databases lacking detailed dimension scores, experts provided ratings based on condition descriptions and overall MOS scores.
  • Newly Generated Database (IS19TRAIN): This database was created with 240 conditions, including various codecs (G.711, G.722, AMR-NB, AMR-WB, EVS, Opus), packet loss, and different types of background noise.
  • Test Databases: Specific databases (TUBDIS, SWISS501, SWISS502, SWISS503) were used for testing, some of which were part of the ITU-T P.AMD pool and had subjective quality dimension scores.

Key Insights and Ideas

  • Perceptual Dimensions as Diagnostic Tools: The core idea is that by estimating specific perceptual dimensions (noisiness, coloration, discontinuity), we gain actionable insights into the root causes of speech quality degradation, moving beyond a single "bad" or "good" score.
  • Deep Learning for Granular Quality Assessment: The paper demonstrates that deep learning, specifically CNN-LSTM architectures, is effective not only for overall quality prediction but also for decomposing quality into its constituent perceptual elements.
  • The Power of Separate Models: Training individual BiLSTM models for each dimension proved more effective than a single multi-output model, highlighting the benefit of tailoring the learning process to specific perceptual attributes.
  • Robustness through Data Augmentation: Adding white noise to training signals and normalizing database biases (e.g., setting clean reference MOS to 4.75) are crucial steps for improving model robustness and generalization.

Key Results

The proposed single-ended model achieved promising results:

  • Average RMSE* of 0.24 across all dimensions and test databases, which is well below the ITU-T performance objective of 0.35.
  • Worst-case RMSE* of 0.35, also meeting the ITU-T requirement of 0.5 for unknown test databases.
  • The model outperforms the intrusive model DIAL (which requires a reference signal) on average for all three quality dimensions.
  • The lowest error was observed for Coloration (RMSE* of 0.19), while Noisiness and Discontinuity had slightly higher errors (RMSE* of 0.26 on average), but still well within performance objectives.
  • Correlation diagrams show good agreement between predicted and subjective MOS values, though some databases exhibited lower correlations due to unbalanced quality distributions.

Limitations and Future Work

  • Reliance on Expert Scores: While efforts were made to annotate databases, some relied on expert ratings rather than full subjective listening tests, which can introduce bias.
  • Generalization to Spontaneous Speech: The current model was trained and tested on recorded speech. The authors plan to investigate its performance on spontaneous speech, which is more representative of real-world single-ended scenarios.
  • Further Subjective Experiments: The authors intend to conduct more subjective experiments to refine the model and reduce reliance on expert scores.
  • Exploring Other Deep Learning Architectures: While CNN-LSTM proved effective, further exploration of other deep learning architectures could potentially yield even better results.

Summary updated

BibTeX
@inproceedings{mittag2019quality,
  author = {Gabriel Mittag and Sebastian Möller},
  title = {{Quality Degradation Diagnosis for Voice Networks — Estimating the Perceived Noisiness, Coloration, and Discontinuity of Transmitted Speech}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure