A DNN-HMM Approach to Non-Negative Matrix Factorization Based Speech Enhancement

Ziteng Wang, Xu Li, Xiaofei Wang, Qiang Fu, Yonghong Yan

Introduction

Non-negative Matrix Factorization (NMF) is a fundamental technique in speech enhancement, where audio signals are decomposed into a dictionary of basis patterns and their corresponding weights. In speaker-independent scenarios, researchers often utilize a large collection of "local dictionaries" to represent diverse human speech sounds. However, selecting the optimal dictionary for a specific noisy audio block remains a significant challenge.

Traditional methods rely on mathematical sparsity penalties (such as the L1 norm) to select patterns, but these often ignore temporal continuity. This neglect can lead to "musical noise" or artifacts caused by erratic switching between dictionary states. This paper proposes a hybrid DNN-HMM framework designed to improve model selection by ensuring both local accuracy and temporal consistency.

Methodology

The proposed system follows a three-stage hybrid approach to enhance noisy speech:

  1. Clustering & Dictionary Learning: Clean speech is clustered into 20 distinct spectral states using K-means. A local NMF dictionary is then learned for each cluster.
  2. DNN Classification: A Deep Neural Network (DNN) is trained to predict the probability that a noisy speech block belongs to each of the 20 states, replacing traditional static penalty terms with learned logic.
  3. HMM & Viterbi Decoding: To prevent rapid "flickering" between states, a first-order Hidden Markov Model (HMM) defines transition probabilities. The Viterbi algorithm identifies the most likely sequence of dictionaries over time, effectively smoothing the transitions.
  4. Reconstruction: Speech is reconstructed using Wiener-type filtering based on the selected optimal dictionary sequence.
The hybrid DNN-HMM architecture illustrates how HMM states (s) govern the selection of DNN-based features over time.

Experimental Setup

  • Speech Data: The TIMIT corpus was used, with 20 male speakers for training and 5 different speakers for testing to ensure speaker independence.
  • Noise Conditions: 10 non-stationary noise types (e.g., keyboard typing, machine guns) were mixed at 0dB SNR.
  • Unsupervised Testing: 100 non-speech environmental sounds were used to test the model's ability to generalize to unseen noise.

Key Insights and Results

The integration of the HMM provides a critical "smoothing" effect. While a DNN-only approach might be confused by sudden bursts of noise, the HMM's transition probabilities act as a temporal anchor, maintaining the correct speech state.

Spectrogram analysis (a-d) and state tracking (e) demonstrate the proposed method's ability to reconstruct clean speech and maintain smooth state transitions compared to noisy baselines.

Quantitatively, the DNN-HMM method outperformed several baselines:

  • Signal-to-Distortion Ratio (SDR): The proposed method reached 9.07 dB, significantly higher than the Mixture of Local Dictionaries (MLD) baseline of 8.28 dB and the noisy baseline of 0.00 dB.
  • Perceptual Quality (PESQ): The system scored 2.38, compared to 2.17 for MLD and 1.90 for Log-MMSE.
  • Oracle Performance: An "oracle test" (using true labels) yielded 9.37 dB, suggesting the proposed system is approaching the theoretical performance ceiling for this architecture.

Limitations and Future Work

Despite its success, the model showed a performance drop of approximately 1.4 dB when encountering noise types not present in the training set, indicating sensitivity in the DNN component. Future research may explore "relaxing" the single-dictionary constraint to allow for dictionary blending, or implementing higher-order HMMs to capture longer-term temporal dependencies and speech rhythms.

Summary updated

Abstract

General speaker-independent models have been used in non-negative matrix factorization (NMF) based speech enhancement algorithms for the practical applicability. And additional regulation is necessary when choosing the optimal models for speech reconstruction. In this paper, we propose a novel utilization of deep neural network (DNN) to select the models used for separating speech from noise. Specifically, multiple local dictionaries are learned, whereas only one is activated for each block in the separation step. Besides, the temporal dependencies between blocks are represented by hidden Markov model (HMM), with which it turns out a hybrid DNN-HMM framework. The most probable activation sequence is then solved by the Viterbi algorithm. Experimental evaluations which focus on a speech denoising application are carried out. The results confirm that our proposed approach achieves better performance when compared with some existing methods.
BibTeX
@inproceedings{wang2016dnn,
  author = {Ziteng Wang and Xu Li and Xiaofei Wang and Qiang Fu and Yonghong Yan},
  title = {{A DNN-HMM Approach to Non-Negative Matrix Factorization Based Speech Enhancement}},
  booktitle = {Interspeech},
  year = {2016},
  doi = {10.21437/interspeech.2016-147},
}
Figure