Supervised Classifiers for Audio Impairments with Noisy Labels

Chandan K. Reddy, Ross Cutler, Johannes Gehrke

Abstract

Voice-over-Internet-Protocol (VoIP) calls are prone to various speech impairments due to environmental and network conditions resulting in bad user experience. A reliable audio impairment classifier helps to identify the cause for bad audio quality. The user feedback after the call can act as the ground truth labels for training a supervised classifier on a large audio dataset. However, the labels are noisy as most of the users lack the expertise to precisely articulate the impairment in the perceived speech. In this paper, we analyze the effects of massive noise in labels in training dense networks and Convolutional Neural Networks (CNN) using engineered features, spectrograms and raw audio samples as inputs. We demonstrate that CNN can generalize better on the training data with a large number of noisy labels and gives remarkably higher test performance. The classifiers were trained both on randomly generated label noise and the label noise introduced by human errors. We also show that training with noisy labels requires a significant increase in the training dataset size, which is in proportion to the amount of noise in the labels.

Introduction

In VoIP services like Skype or Microsoft Teams, identifying the root cause of poor audio quality—whether it be background noise, network-induced speech distortion, or low volume—is essential for engineering improvements. While user-provided ratings offer a massive stream of data, this feedback is inherently "noisy." Non-expert users frequently misidentify technical impairments, leading to inaccurate ground truth labels. This research investigates whether reliable supervised classifiers can be built from such inaccurate data by comparing different neural network architectures and evaluating their robustness against varying levels of label error.

Methodology

The study evaluates three distinct architectural approaches for impairment classification:

  1. Engineered Features + Dense Network: A standard fully connected network utilizing 18 traditional signal processing features, such as Signal-to-Noise Ratio (SNR) and spectral centroid.
  2. Log Mel Spectrogram + 2D-CNN: A 2D Convolutional Neural Network (CNN) that processes visual representations of audio frequencies over time.
  3. Raw Audio + 1D-CNN: A 1D-CNN that performs automated feature extraction directly from raw waveforms.
Architectural comparison of (a) Dense Network, (b) 2D-CNN, and (c) 1D-CNN models.

To simulate real-world conditions, the models were tested against two types of label noise: Synthetic Noise (randomly flipping labels at 20%, 50%, and 80% error rates) and Real Human Noise (labels from non-expert workers, resulting in a 34% error rate).

Experimental Setup

The researchers utilized a controlled dataset of 5,000 synthesized audio clips across five categories: Clean Speech, Background Noise (office, car, restaurant), Reverberation, Speech Distortion (simulated packet loss), and Low Volume. To test the limits of label noise robustness, the training sets were expanded up to 600,000 examples.

Results and Discussion

CNN Robustness to Noise

The most significant finding is the remarkable robustness of CNN architectures. While the performance of dense networks using engineered features collapsed as noise increased, the 2D-CNN maintained high accuracy. Even at a staggering 80% label error rate, the 2D-CNN achieved 89% accuracy on a clean test set, compared to just 35% for the engineered features model. This suggests that the spatial and temporal patterns learned by CNNs are inherently more resistant to contradictory labels.

The Data-Noise Trade-off

There is a clear non-linear relationship between label noise and the volume of data required for model convergence. As label unreliability increases, the training dataset must grow exponentially to maintain high accuracy thresholds.

The impact of label noise on the required training dataset size to achieve specific accuracy targets (75%, 80%, and 85%).

Human Error Patterns

Analysis of human-labeled data revealed that non-experts struggle specifically with the "No Impairment" category, often incorrectly flagging clean audio as having low volume or background noise. Despite this 34% human error rate, the 2D-CNN achieved a high real-world performance of 91% accuracy.

Conclusion

Convolutional Neural Networks, particularly those utilizing Log Mel Spectrograms, provide a robust solution for training audio impairment classifiers on unreliable user feedback. By scaling dataset size in proportion to label noise, it is possible to build highly accurate monitoring systems that are computationally efficient enough for real-time VoIP applications. Future work will focus on expanding the model to handle overlapping impairments and exploring self-supervised learning to further reduce reliance on human labeling.

Summary updated

Abstract

Voice-over-Internet-Protocol (VoIP) calls are prone to various speech impairments due to environmental and network conditions resulting in bad user experience. A reliable audio impairment classifier helps to identify the cause for bad audio quality. The user feedback after the call can act as the ground truth labels for training a supervised classifier on a large audio dataset. However, the labels are noisy as most of the users lack the expertise to precisely articulate the impairment in the perceived speech. In this paper, we analyze the effects of massive noise in labels in training dense networks and Convolutional Neural Networks (CNN) using engineered features, spectrograms and raw audio samples as inputs. We demonstrate that CNN can generalize better on the training data with a large number of noisy labels and gives remarkably higher test performance. The classifiers were trained both on randomly generated label noise and the label noise introduced by human errors. We also show that training with noisy labels requires a significant increase in the training dataset size, which is in proportion to the amount of noise in the labels.
BibTeX
@inproceedings{reddy2019supervised,
  author = {Chandan K. Reddy and Ross Cutler and Johannes Gehrke},
  title = {{Supervised Classifiers for Audio Impairments with Noisy Labels}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure