Hidden Markov models as priors for regularized nonnegative matrix factorization in single-channel source separation

Emad M. Grais, Hakan ErdoganInterspeech 2012

TL;DR

This paper introduces a novel method for separating mixed audio signals (like speech and music) from a single channel. They achieve this by combining Nonnegative Matrix Factorization (NMF) with Hidden Markov Models (HMMs) as statistical "priors" to guide the separation process, leading to improved accuracy.

What Problem Does This Paper Address?

Separating individual sound sources from a single mixed audio recording (single-channel source separation or SCSS) is a challenging problem. Imagine trying to isolate a single voice from a noisy party or a specific instrument from an ensemble. Traditional methods like Nonnegative Matrix Factorization (NMF) work by decomposing the audio's spectral representation into "basis" components (like the characteristic sounds of speech or music) and "gain" components (how much of each basis is present at any given time). However, NMF alone can struggle to accurately disentangle sources, especially when they have similar spectral characteristics or when the mixture is complex. This paper aims to improve SCSS by incorporating richer statistical information about how the "gains" for a specific source should behave over time, leveraging the sequential nature of audio.

Methods

The core of this work is the integration of Hidden Markov Models (HMMs) as priors into the NMF framework for SCSS. Here's a breakdown:

  • NMF for Source Separation: The paper uses NMF to decompose the spectrogram of the mixed signal. The idea is that the mixed signal's spectrogram can be approximated as a weighted sum of learned basis spectrograms for each source (e.g., speech basis, music basis). The weights are the "gain" matrices.
  • Training Basis Matrices: For each source type (speech, music), NMF is used on training data to learn a "basis matrix" (representing the characteristic spectral patterns) and a "gain matrix" (representing how these patterns combine over time).
  • HMMs as Gain Priors: Crucially, the gain matrices from the training data are used to train HMMs. These HMMs learn the typical temporal sequences of gains for each source. The gains are log-transformed and normalized to make the HMM priors energy-independent, meaning they can work with signals of varying loudness.
  • Regularized NMF: During the separation of a new mixed signal, the NMF process is modified. Instead of just minimizing the reconstruction error (how well the separated sources add up to the original mix), it also maximizes the log-likelihood of the estimated gain sequences with respect to their corresponding trained HMM priors. This is achieved by adding a regularization term to the NMF cost function.
  • Optimization: The paper details how to derive multiplicative update rules for the gain matrix that simultaneously minimize the NMF reconstruction error and maximize the HMM likelihood, effectively balancing the two objectives.
  • Wiener Filtering: After obtaining the estimated gain matrices and using the trained basis matrices, an initial estimate of each source's spectrogram is produced. This is then refined using Wiener filtering to produce the final source separation.

Datasets

  • Training Data:
    • Speech: 540 short utterances from a single speaker.
  • Music: Piano music from the piano society website (12 pieces from different composers by a single artist).
  • Testing Data:
    • Speech: 20 utterances from the same speaker used for training.
  • Music: One piece of piano music left out from the training set.
  • Mixture Generation: Test speech and music files were mixed at various Speech-to-Music Ratio (SMR) levels (in dB) to create test utterances.

Key Insights and Ideas

  • Leveraging Temporal Structure: The key insight is that the temporal evolution of gains (how loud a source is over time) carries significant information that can be exploited. HMMs are well-suited to model these sequential patterns.
  • Energy Independence: Normalizing the gains before training HMMs is crucial. It allows the prior models to be effective regardless of the absolute loudness of the source signals in the mixed audio, making the system more robust.
  • Balancing Reconstruction and Prior Knowledge: The regularization approach elegantly balances two competing goals: accurately reconstructing the mixed signal (NMF's strength) and ensuring the separated sources conform to expected temporal patterns (HMM's strength).
  • Improved Separation Accuracy: By guiding the NMF with HMM priors, the algorithm can make more informed decisions about which spectral components belong to which source, leading to cleaner separations.

Key Results

  • The proposed regularized NMF method, incorporating HMM gain priors, consistently outperforms standard NMF without priors across various Speech-to-Music Ratio (SMR) levels.
  • For instance, at an SMR of 20 dB, the Signal-to-Noise Ratio (SNR) improved from 17.26 dB (NMF without prior) to 21.87 dB with the proposed method and carefully chosen regularization parameters.
  • The paper demonstrates that the choice of regularization parameters (λ for speech and music) is important and can be tuned based on the expected SMR of the mixed signal for optimal results. For lower SMRs (less speech energy), a slightly higher λ for speech is beneficial, indicating reliance on the prior.

Limitations and Future Work

  • Parameter Tuning: The performance is sensitive to the choice of regularization parameters (λ). While the paper shows how to tune them for better results, finding optimal values automatically for unknown mixtures remains an area for exploration.
  • Number of States and Components: The number of states (Q) in the HMM and the number of Gaussian mixture components (K) in the emission distributions are fixed parameters. Determining optimal values for these based on the data could further improve performance.
  • Computational Cost: While not explicitly detailed, incorporating HMM likelihood calculations into the NMF optimization likely increases the computational complexity compared to standard NMF.
  • Future Work: The authors suggest that exploring different prior models or incorporating other types of prior information could lead to further improvements in source separation. They also imply that the energy-independent nature of the priors is a significant advantage for real-world applications.

Summary updated

BibTeX
@inproceedings{grais2012hidden,
  author = {Emad M. Grais and Hakan Erdogan},
  title = {{Hidden Markov models as priors for regularized nonnegative matrix factorization in single-channel source separation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure