Effect of speech priors in single-channel speech-music separation for ASR

Cemil Demir, A. Taylan Cemgil, Murat SaraçlarInterspeech 2012

TL;DR

This paper enhances single-channel speech-music separation for Automatic Speech Recognition (ASR) by incorporating prior knowledge about speech characteristics. By using a more sophisticated probabilistic model (complex Gaussian observation model with Inverse-Gamma priors), they achieve improved separation performance, leading to better ASR accuracy, especially in noisy broadcast environments.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle when background music is present in audio, leading to a significant drop in performance. This is a common issue in broadcast news and other media. While separating speech from music in a single audio channel is challenging, this paper focuses on improving existing methods by leveraging prior information about speech to make the separation process more robust and effective. The goal is to clean up the audio signal before it's fed into an ASR system, thereby improving the ASR's accuracy.

Methods

The core contribution is the extension of a previously proposed "catalog-based" speech-music separation method. This method relies on a "jingle catalog" – a collection of repetitive music segments often found in broadcasts. The key innovation here is the incorporation of a prior speech model.

Instead of just modeling the music, the authors introduce a probabilistic model for the speech component. They adopt a complex Gaussian observation model for both speech and music signals. This model is mathematically equivalent to minimizing the Itakura-Saito (IS) divergence, which is a measure of difference between power spectrograms. For the speech component, they use an Inverse-Gamma prior on the template matrix, chosen for its conjugacy property with the complex Gaussian distribution. This allows for a more principled way to incorporate prior speech information.

To handle the complexity of estimating all the unknown parameters and latent variables, they employ variational inference. This technique approximates the true posterior distribution with a simpler, factorized one, enabling efficient computation of updates for the model parameters. They also explore different gain estimation strategies, including a novel Inverse-Gamma-Markov-Chain (IGMC) approach to smooth out abrupt changes in estimated gain values over time, which is crucial for realistic audio signals.

Datasets

The paper uses a combination of synthesized and real-world data:

  • Speech Training Data: Three types of speech datasets are used to train the prior speech model: "Self" (same speaker), "All" (including the speaker), and "Other" (excluding the speaker). These are used to train a CMU-Sphinx HMM-based continuous density speech recognizer.
  • Test Set: A synthetic test set is created by mixing Turkish broadcast news speech with 10 different jingles at a 10dB Speech-to-Music Ratio (SMR). The jingles are sourced from real broadcast news.

Key Insights and Ideas

  • Prior Speech Information is Beneficial: The most significant insight is that explicitly modeling prior speech characteristics, beyond just modeling the music, demonstrably improves speech-music separation performance. This is evident across various evaluation metrics.
  • Complex Gaussian Model Advantage: Using a complex Gaussian observation model (and thus minimizing IS divergence) generally leads to better separation results compared to the Poisson model (KL divergence) previously used, especially when combined with prior speech information.
  • Gain Estimation Matters: The choice of gain estimation strategy significantly impacts performance. The proposed IGMC approach, which enforces temporal smoothness, shows promise, though its benefits vary depending on the divergence measure used.
  • Robustness to Speaker Variability: The "Other" speech training data type (excluding the target speaker) performs comparably to "Self" and "All" types. This is a crucial finding, suggesting that the method can be effective even when the exact speaker in the audio is not present in the training data, making it more practical for real-world applications.

Key Results

The paper presents quantitative results in terms of Signal-to-Music Ratio (SMR), Source-to-Artifact Ratio (SAR), and Word Error Rate (WER) for the ASR system.

  • SMR Improvement: Incorporating prior speech models consistently increases SMR values across different divergence measures and gain estimation strategies. For example, with the original gain estimation, SMR increases from 25.9 dB (no prior) to around 28.5-28.9 dB (with priors) for KL divergence, and similarly for IS divergence.
  • SAR Improvement: Prior speech information also leads to higher SAR values, indicating better preservation of the speech signal while reducing music artifacts.
  • WER Reduction: The ultimate goal, improved ASR performance, is achieved. The WER drops significantly when using the separation method with prior speech models. For instance, the WER for mixed speech is 83.2%, while with separation methods using priors, it can drop to around 31.7-35.5% (depending on the specific method and parameters). The paper highlights that using prior speech models generally improves WER more than the GMC gain estimation strategy in the IS case, and vice-versa in the KL case, but overall, priors are beneficial.

Limitations and Future Work

  • Jingle Identity Assumption: The current method assumes the identity of the jingle is known a priori. This is a significant simplification for real-world scenarios where jingle identification might be part of the problem.
  • Synthetic Mixing: The test set is created by synthetically mixing speech and music. While controlled, this might not perfectly replicate the complexities of real-world mixed audio.
  • Gain Estimation Strategy Performance: While IGMC is introduced, its relative improvement over the original strategy varies, suggesting further research into optimal gain estimation is needed.
  • Future Work: The authors suggest exploring semi-supervised approaches and more advanced methods for jingle identification to make the system more practical. Further investigation into the interplay between divergence measures, gain estimation, and prior speech models is also implied.

Summary updated

BibTeX
@inproceedings{demir2012effect,
  author = {Cemil Demir and A. Taylan Cemgil and Murat Saraçlar},
  title = {{Effect of speech priors in single-channel speech-music separation for ASR}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure