Practical Applicability of Deep Neural Networks for Overlapping Speaker Separation

Pieter Appeltans, Jeroen Zegers, Hugo Van hammeInterspeech 2019

TL;DR

This paper investigates the real-world applicability of deep learning models for separating overlapping speech. The authors demonstrate that these models generalize well across different languages and can be adapted to perform better in noisy environments, offering practical insights for deploying speaker separation technology.

What Problem Does This Paper Address?

Imagine you're in a noisy room with multiple people talking at once. Your brain is remarkably good at focusing on one voice, filtering out the others. However, for machines, this "cocktail party problem" is incredibly challenging. Specifically, this paper tackles the problem of overlapping speaker separation, which aims to isolate individual speech signals from a mixed audio recording where multiple people are speaking simultaneously. While deep learning methods have shown promise, their effectiveness in diverse, real-world scenarios (like different languages and background noise) remains an open question. This research explores how well these advanced techniques perform when faced with these practical challenges.

Methods

The paper focuses on two deep learning architectures for speaker separation:

  • Deep Clustering (DC): This method uses a recurrent neural network (RNN) with Long Short-Term Memory (LSTM) cells to map segments of the audio spectrogram to an "embedding vector." The network is trained to ensure that embeddings from segments dominated by the same speaker are close together, while embeddings from different speakers are far apart. After training, K-means clustering is applied to these embeddings to identify distinct speakers, and then masks are generated to separate them.
  • Deep Attractor Networks (DAN): Similar to DC, DAN also uses an RNN with LSTMs to generate embedding vectors. However, instead of just clustering, DAN learns to map embeddings to "attraction points" in the embedding space, with each point representing a speaker. This approach aims to pull embeddings associated with a specific speaker towards their designated attraction point.

The paper also proposes modifications to these methods to improve their performance in noisy conditions. These modifications involve adding a "noise mask" output to the network, which helps to suppress background noise before or during the speaker separation process.

Datasets

The experiments utilize several datasets:

  • Global Phone Corpus [15]: Used for generating mixtures of speakers in various languages (Arabic, French, Mandarin, Portuguese, Spanish, Swedish) for training and testing.
  • Wall Street Journal Database [18]: Used for obtaining speech signals.
  • Third CHiME Speech Separation and Recognition Challenge dataset [19]: Used for realistic background noise samples.

The authors created new experimental setups by combining these sources to test language generalization and noise robustness.

Key Insights and Ideas

  • Language Generalization: Deep learning models for speaker separation are not limited to the language they were trained on. They can generalize to other languages, though performance might slightly degrade for untrained languages.
  • The Power of Relatedness: The degree of performance degradation when generalizing to a new language seems to depend on how linguistically related the training and testing languages are.
  • Noise Robustness is Achievable: With modifications, deep learning models can effectively handle realistic background noise, significantly improving separation quality.
  • Mandarin as a Special Case: Mandarin, being a tonal language, showed the best performance for both DC and DAN when trained and tested on the same language. This suggests that tonal information might be a useful cue for speaker separation, though more research is needed.
  • Beyond Phonetics: The models seem to exploit cues beyond simple phonetic or phonotactic information, suggesting they capture more complex aspects of speech for separation.

Key Results

  • Cross-Lingual Performance: When trained on one language and tested on another, performance degradation was observed, but it was generally manageable (around 1 dB for some language pairs). For Mandarin, the drop was more significant (around 4 dB).
  • Multi-Language Training: Training with multiple languages (e.g., French and Turkish) generally improved performance for untrained languages compared to training with a single language. However, for trained languages, using multiple languages sometimes led to a slight decrease in performance.
  • Noise Handling:
    • Models trained without noise performed poorly on noisy data (negative SDR scores).
  • Treating noise as a third speaker during training improved performance.
  • The proposed modified DC and DAN methods achieved the best results in noisy conditions, with SDR scores around 5-7 dB in challenging scenarios. For instance, the modified DAN achieved an SDR of 7.33 dB when noise was 3dB quieter than speakers.

Limitations and Future Work

  • Untrained Noise: The current experiments on noise robustness primarily used noise types that were present in the training data. The authors acknowledge that testing with completely "untrained" noise types is a limitation.
  • Language Specificity: While generalization is good, the significant drop in performance for Mandarin suggests that certain language-specific features (like tonality) might be exploited by the models, and further investigation into this is warranted.
  • Future Work: The authors suggest exploring the performance of these methods on "untrained" noise types and further investigating the role of language-specific cues in speaker separation.

Summary updated

BibTeX
@inproceedings{appeltans2019practical,
  author = {Pieter Appeltans and Jeroen Zegers and Hugo Van hamme},
  title = {{Practical Applicability of Deep Neural Networks for Overlapping Speaker Separation}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure