Supervised Classifiers for Audio Impairments with Noisy Labels

Chandan K.a. Reddy, Ross Cutler, Johannes Gehrke

TL;DR

This paper explores training audio impairment classifiers using noisy labels, a common issue in real-world user feedback. They found that Convolutional Neural Networks (CNNs), particularly those using Log Mel Spectrograms, can achieve high accuracy even with significant label noise, demonstrating their robustness and potential for practical applications.

What Problem Does This Paper Address?

In voice-over-IP (VoIP) calls, various audio impairments like background noise, reverberation, speech distortion, and low volume can degrade user experience. To fix these issues, it's crucial to identify their causes. While user feedback can provide "ground truth" labels for training automated classifiers, this feedback is often noisy. Users may lack the expertise to accurately describe the specific impairment, leading to unreliable labels. This paper tackles the challenge of training effective audio impairment classifiers when faced with this pervasive label noise, aiming to bridge the gap between synthetic, perfectly labeled data and real-world, imperfect user feedback.

Methods

The authors investigate three supervised classification approaches:

  • Engineered Features with Dense Network: This method uses 18 hand-crafted signal processing features (e.g., spectral centroid, zero-crossing rate) as input to a fully connected neural network. This approach is computationally efficient but relies heavily on prior signal processing knowledge.
  • Log Mel Spectrogram with 2D CNN: Here, audio is converted into Log Mel Spectrograms, which are then fed into a 2D Convolutional Neural Network (CNN). This approach captures more spectral information and leverages the power of CNNs for pattern recognition.
  • Raw Audio Samples with 1D CNN: The raw audio waveform is directly used as input to a 1D CNN. This method avoids any information loss from feature extraction or spectral transformations, allowing the network to learn features directly from the signal.

The core novelty lies in evaluating these architectures under varying degrees of label noise, including both synthetically generated noise and real-world noise from human annotators.

Datasets

The primary dataset used was synthesized, comprising 5000 audio clips covering four impairment classes (background noise, reverberation, speech distortion, low volume) and a "no impairment" class. Each clip is approximately 10 seconds long.

For labeling, the authors employed Microsoft's Universal Human Relevance System (UHRS), a crowdsourcing platform similar to Amazon Mechanical Turk. This platform was used to collect labels from non-expert human judges, introducing real-world label noise. The paper also discusses experiments with synthetically generated label noise at error rates of 20%, 50%, and 80%, as well as scenarios with a fixed number of clean labels and varying numbers of noisy labels (2 or 8 noisy labels per clean label).

Key Insights and Ideas

  • CNNs are Robust to Label Noise: The most significant finding is that CNN architectures (both 2D and 1D) demonstrate remarkable resilience to noisy labels. They can generalize well and maintain high accuracy even when trained on data with substantial label errors.
  • Log Mel Spectrograms Shine: Among the tested approaches, the Log Mel Spectrogram input with a 2D CNN consistently performed best, especially under noisy label conditions. This suggests that this representation strikes a good balance between capturing relevant audio characteristics and being amenable to CNN processing.
  • Raw Audio is Promising but Demanding: While using raw audio samples with a 1D CNN also shows good performance, it requires significantly larger training datasets and more computational resources due to the high dimensionality of the input.
  • More Noise Requires More Data: The paper quantifies the relationship between label noise and the required training dataset size. As the proportion of noisy labels increases, the amount of training data needed to achieve a target accuracy grows exponentially.
  • Human Noise is Complex: Real-world human-generated noise (from UHRS) is not simply random. The paper notes that the "No Impairment" class was particularly prone to misclassification, as human judges sometimes incorrectly labeled clips with background noise or low volume as "no impairment."

Key Results

  • A Log Mel Spectrogram + 2D CNN classifier achieved approximately 90% accuracy even when trained on labels with an 80% error rate.
  • When trained on UHRS labels (found to be approximately 66% accurate), the Log Mel Spectrogram + 2D CNN achieved 91% accuracy, demonstrating its effectiveness with real-world noisy data.
  • The engineered features approach, while efficient, saw a more significant drop in accuracy (down to 35% at 80% error rate) compared to the CNN-based methods.
  • The training dataset size required to achieve a 75% accuracy threshold increased dramatically with higher noise levels, showing an exponential relationship.

Limitations and Future Work

  • Dataset Size vs. Noise: The paper highlights that while CNNs are robust, a sufficient number of clean labels are still crucial. The performance degrades significantly if the proportion of clean labels becomes too small.
  • Noise Distribution Matters: The authors acknowledge that the performance can depend on the specific distribution of noise in the labels, which can be complex in real-world scenarios.
  • Future Work: The authors suggest that further research could explore more advanced noise-robust training techniques and investigate the impact of different noise distributions on classifier performance. They also imply that the robustness of these models could significantly reduce false positives in large-scale live systems.

Summary updated

Abstract

Voice-over-Internet-Protocol (VoIP) calls are prone to various speech impairments due to environmental and network conditions resulting in bad user experience. A reliable audio impairment classifier helps to identify the cause for bad audio quality. The user feedback after the call can act as the ground truth labels for training a supervised classifier on a large audio dataset. However, the labels are noisy as most of the users lack the expertise to precisely articulate the impairment in the perceived speech. In this paper, we analyze the effects of massive noise in labels in training dense networks and Convolutional Neural Networks (CNN) using engineered features, spectrograms and raw audio samples as inputs. We demonstrate that CNN can generalize better on the training data with a large number of noisy labels and gives remarkably higher test performance. The classifiers were trained both on randomly generated label noise and the label noise introduced by human errors. We also show that training with noisy labels requires a significant increase in the training dataset size, which is in proportion to the amount of noise in the labels.
BibTeX
@inproceedings{reddy2019supervised,
  author = {Chandan K.A. Reddy and Ross Cutler and Johannes Gehrke},
  title = {{Supervised Classifiers for Audio Impairments with Noisy Labels}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure