Towards Adapting NMF Dictionaries Using Total Variability Modeling for Noise-Robust Acoustic Features

Kunal Dhawan, Colin Vaz, Ruchir Travadi, Shrikanth S. NarayananarXiv 2019 · arXiv

Introduction

Automatic Speech Recognition (ASR) systems frequently encounter performance degradation when deployed in real-world environments characterized by background noise and varying channel conditions. Traditional preprocessing techniques, such as Wiener filtering, often introduce artifacts like "musical noise" that can be more detrimental to ASR accuracy than the original interference. While Non-negative Matrix Factorization (NMF) has emerged as a powerful tool for feature extraction, many existing implementations rely on parallel corpora—clean and noisy versions of the same speech—which are difficult and expensive to acquire. This paper proposes a novel framework that combines NMF with Total Variability Modeling (TVM) to adapt acoustic dictionaries to individual utterances without the need for parallel training data.

Proposed Methodology

The core of the proposed approach is a three-stage pipeline designed to extract robust acoustic features by modeling how spectral dictionaries shift in response to environmental variability.

1. Universal Background Model (UBM)

Rather than training on pristine clean speech, the system utilizes NMF to learn a "universal" dictionary from a large, heterogeneous set of noisy speech recordings. This dictionary captures the broad spectral components present across diverse acoustic environments.

2. Total Variability Modeling

Borrowing from the "i-vector" framework used in speaker verification, the authors assume that any specific utterance's dictionary is a shifted version of the UBM dictionary. They train a total variability matrix ($T$) to learn the subspace of typical dictionary variations caused by noise, speaker characteristics, and channel effects.

NMF decomposition of a magnitude spectrogram into a dictionary (W) and activation matrix (H).

3. Utterance-Specific Adaptation

During inference, a low-dimensional "i-vector" is calculated for each new noisy utterance. This vector describes the specific acoustic environment and is used to adapt the UBM dictionary to the current recording. The final acoustic features for the ASR system are the log-activations ($H$) derived from this customized dictionary.

Experimental Evaluation

The method was evaluated using the Aurora 4 corpus (based on Wall Street Journal data) and the DEMAND database of real-world noise (e.g., parks, restaurants). The training set consisted of 7,138 utterances corrupted with six specific noises at 5–15 dB SNR. Testing was conducted on both "seen" noises (included in training) and "unseen" noises (e.g., hallways, stations) to evaluate generalization capabilities.

Results and Discussion

The proposed method was compared against standard baselines, including Mel-Frequency Cepstral Coefficients (MFCC), Power-Normalized Cepstral Coefficients (PNCC), and Convolutive NMF (CNMF).

Word Error Rate (WER) comparison across different noise conditions for the proposed and baseline methods.

While the proposed method did not outperform the baselines in terms of raw Word Error Rate (WER) on clean speech, it demonstrated significant stability. A key finding was the smaller performance gap between clean and unseen noise conditions compared to traditional features. Interestingly, performance on "unseen" noise was consistently better than on "seen" noise across all methods; the authors attribute this to the high non-stationarity and background speech present in the "seen" training noises (like restaurant environments), which are inherently more challenging for ASR.

Conclusion and Future Work

The primary contribution of this work is demonstrating that Total Variability Modeling can achieve noise robustness in NMF-based features without requiring parallel training data. The current performance limitations are likely due to the relatively small training set (7k utterances), which may be insufficient to fully define the variability subspace ($T$). Future research will focus on scaling the training to larger datasets like LibriSpeech and incorporating more explicit modeling of channel variability to improve real-world performance.

Summary updated

Abstract

We propose an algorithm to extract noise-robust acoustic features from noisy speech. We use Total Variability Modeling in combination with Non-negative Matrix Factorization (NMF) to learn a total variability subspace and adapt NMF dictionaries for each utterance. Unlike several other approaches for extracting noise-robust features, our algorithm does not require a training corpus of parallel clean and noisy speech. Furthermore, the proposed features are produced by an utterance-specific transform, allowing the features to be robust to the noise occurring in each utterance. Preliminary results on the Aurora 4 + DEMAND noise corpus show that our proposed features perform comparably to baseline acoustic features, including features calculated from a convolutive NMF (CNMF) model. Moreover, on unseen noises, our proposed features gives the most similar word error rate to clean speech compared to the baseline features.
BibTeX
@misc{dhawan2019towards,
  author = {Kunal Dhawan and Colin Vaz and Ruchir Travadi and Shrikanth S. Narayanan},
  title = {{Towards Adapting NMF Dictionaries Using Total Variability Modeling for Noise-Robust Acoustic Features}},
  year = {2019},
  eprint = {1907.06859},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.06859},
}
Figure