Impairments are Clustered in Latents of Deep Neural Network-based Speech Quality Models

Fredrik Cumlin, Xinyu Liang, Victor Ungureanu, Chandan K. Reddy, Christian Schüldt, Saikat Chatterjee

TL;DR

Deep neural networks trained to predict speech quality (how "good" a clip sounds) naturally learn to identify what is wrong with the audio, even though they aren't trained to do so. This research shows that these models organize different types of audio damage—like background noise, clipping, or reverb—into distinct clusters within their internal "latent" representations, and that better quality predictors are also better at identifying specific impairments.

What Problem Does This Paper Address?

In the world of audio processing, Speech Quality Assessment (SQA) models are designed to act like a human listener, providing a score (often a Mean Opinion Score, or MOS) that reflects how clear or degraded a speech sample is. However, most modern SQA models are "black boxes"—they might tell you a clip is a 2 out of 5, but they won't tell you if that's because of a barking dog, a bad internet connection (codec artifacts), or an echoey room.

While traditional audio features like MFCCs (Mel-frequency cepstral coefficients) have shown some ability to cluster noise types, the authors wanted to know if modern Deep Neural Network (DNN) SQA models do this more effectively. Specifically, they investigated whether the process of learning to judge "quality" forces a model to internally categorize the cause of the degradation as a necessary byproduct.

Methods

The researchers focused on the internal "latent" space of SQA models—the layer of data just before the final score is calculated.

  1. Model Evolution: They started with DNSMOS, a popular low-complexity SQA model. They then developed an improved version called DNSMOS+. This new model swaps Mel-spectrograms for raw STFT (Short-Time Fourier Transform) inputs, uses SiLU activation functions, and doubles the latent space dimension from 64 to 128.
  2. The "Probing" Task: To see if the models were clustering impairments, the authors froze the trained SQA models and extracted the latent feature vectors for various audio clips. They then used a simple k-Nearest Neighbor (kNN) classifier on these features. If the kNN could accurately identify the impairment type, it proved the SQA model had naturally "clustered" those impairments in its brain.
  3. Visualization: They used Principal Component Analysis (PCA) to flatten the high-dimensional latent space into a 2D map to visually confirm if different distortions (like "Pitch Shift" vs. "Room Simulator") occupied different areas.

Datasets

  • LibriAugmented (New): The authors created this by taking clean speech from LibriSpeech and applying 16 different types of "damage." This included 9 single impairments (noise, clipping, gain transition, low-pass filter, etc.) and 6 combinations (e.g., noise + reverb).
  • ESC-50: A standard dataset for Environmental Sound Classification containing 50 distinct real-world noise types (baby crying, helicopter, etc.). This was used to see if the SQA models could cluster noises they weren't specifically exposed to during training.
  • Labels: The models were trained using "pseudo-labels" from established objective metrics like PESQ and ViSQOL, which simulate human MOS.

Key Insights and Ideas

  • Clustering is an Emergent Behavior: The models were never told "this is noise" or "this is clipping." They were only told "this sounds bad." The fact that they grouped similar impairments together suggests that categorizing the type of damage is an efficient internal strategy for determining the degree of damage.
  • Better Quality Prediction = Better Categorization: There is a direct correlation between SQA accuracy and impairment classification. As the model gets better at predicting the MOS score (DNSMOS+), its internal clusters become sharper and more accurate.
  • Inductive Bias: The results suggest that DNNs have a natural "inductive bias" to organize these impairments. Even a model with random weights has a tiny bit of this ability, but training it on speech quality amplifies this organization significantly.
  • ViSQOL vs. PESQ: The authors found that models trained on ViSQOL labels were more consistent and better at clustering than those trained on PESQ, suggesting ViSQOL might be a better "teacher" for emulating human judgment across diverse distortions.

Key Results

  • Impairment Classification: On the LibriAugmented dataset (16 classes), DNSMOS+ achieved 94% accuracy in identifying the impairment type. For comparison, the original DNSMOS achieved 86%.
  • Real-World Noise: On the ESC-50 dataset (50 noise types), the latent features of DNSMOS+ achieved 54% accuracy. This is nearly triple the performance of traditional MFCC features (18%) and significantly better than random projection (3%).
  • SQA Performance: DNSMOS+ reached a Pearson Correlation Coefficient (PCC) of 0.93, indicating a very high agreement with objective quality metrics.
  • Visual Evidence: PCA plots (see Figure 2 in the paper) show clear, distinct "islands" for different impairment types, confirming that the model's internal map is highly organized.

Limitations and Future Work

  • Noise Suppression Bias: The authors noted that some existing models (like the original DNSMOS) were trained on data already processed by noise suppressors. This might actually reduce the model's ability to cluster noises because the "evidence" of the noise has been partially erased.
  • Future Directions: The researchers suggest exploring why certain training data configurations lead to better clustering. There is also potential to use these findings to create "interpretable" SQA models that not only give a quality score but also provide a "diagnostic" report of what is wrong with the audio.

Summary updated

Abstract

In this article, we provide an experimental observation: Deep neural network (DNN) based speech quality assessment (SQA) models have inherent latent representations where many types of impairments are clustered. While DNN-based SQA models are not trained for impairment classification, our experiments show good impairment classification results in an appropriate SQA latent representation. We investigate the clustering of impairments using various kinds of audio degradations that include different types of noises, waveform clipping, gain transition, pitch shift, compression, reverberation, etc. To visualize the clusters we perform classification of impairments in the SQA-latent representation domain using a standard k-nearest neighbor (kNN) classifier. We also develop a new DNN-based SQA model, named DNSMOS+, to examine whether an improvement in SQA leads to an improvement in impairment classification. The classification accuracy is 94% for LibriAugmented dataset with 16 types of impairments and 54% for ESC-50 dataset with 50 types of real noises.
BibTeX
@misc{cumlin2025impairments,
  author = {Fredrik Cumlin and Xinyu Liang and Victor Ungureanu and Chandan K. Reddy and Christian Schüldt and Saikat Chatterjee},
  title = {{Impairments are Clustered in Latents of Deep Neural Network-based Speech Quality Models}},
  year = {2025},
  eprint = {2504.21528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.21528},
  howpublished = {arXiv},
}
Figure