Non-negative matrix factorization based compensation of music for automatic speech recognition

Bhiksha Raj, Tuomas Virtanen, Sourish Chaudhuri, Rita SinghInterspeech 2010 · DOI

Abstract

Automatic Speech Recognition (ASR) systems often fail in the presence of non-stationary background noise, such as music. This paper proposes a compensation method based on Non-negative Matrix Factorization (NMF) that treats audio as a composition of speech and music spectral components. By utilizing dictionaries of pre-recorded exemplars, the system can effectively "unmix" the signal in real-time. Evaluations using the Wall Street Journal corpus mixed with diverse music genres demonstrate that NMF-based compensation significantly reduces word error rates, particularly when combined with standard adaptation techniques.

Introduction

Background music is a persistent challenge for ASR because its spectral characteristics change rapidly, unlike stationary noises like fan hums. Traditional statistical models often struggle to capture this complexity, leading to distorted "cleaned" speech and high Word Error Rates (WER). This research, presented at INTERSPEECH 2010, addresses this by employing a compositional model that estimates noise at every instant rather than relying on past averages.

Methods

The core of the proposed approach is NMF-based separation. The magnitude spectrum of the noisy audio is modeled as a weighted sum of "basis vectors" or spectral templates.

  1. Exemplar-Based Dictionaries: Instead of learning complex mathematical models, the researchers used random snippets (exemplars) from training data to create separate dictionaries for speech and music.
  2. Signal Unmixing: A noisy signal is reconstructed using a combination of elements from both dictionaries. By minimizing the generalized Kullback-Leibler divergence between the reconstruction and the original signal, the system identifies the optimal weights for each component.
  3. Reconstruction: The music components are discarded, and the speech is reconstructed using only the speech dictionary weights. This "cleaned" audio is then processed by a standard ASR engine (CMU-Sphinx).

Experimental Setup

The system was tested using the Wall Street Journal (WSJ) corpus for speech and the RWC music database for background noise (including Classical, Jazz, Latin, and World music). Acoustic models were trained on the Resource Management database. Audio was mixed at Signal-to-Noise Ratios (SNRs) ranging from -5dB (very noisy) to 20dB (relatively clean).

Results

The NMF method showed massive accuracy gains across all tested scenarios. At 0dB SNR, where speech and music are equally loud, accuracy (100 - WER) improved significantly compared to uncompensated baselines.

Comparison of ASR accuracy across different compensation methods and SNRs.

As shown in the figure above, NMF-based compensation significantly outperforms baseline methods and traditional techniques like Vector Taylor Series (VTS). Furthermore, the method exhibits strong synergy with Maximum Likelihood Linear Regression (MLLR) adaptation, achieving the highest overall performance.

ASR accuracy across different music genres (a-d) showing NMF (red) vs uncompensated baseline (blue).

The improvements were consistent across various music genres, as illustrated in the figure above. While performance was slightly better against classical music due to its distinct spectral structure, the "generic" dictionary approach proved robust even when the specific music style or speaker was unknown.

Conclusion

NMF-based compensation provides a powerful tool for ASR in musically noisy environments. Its ability to perform instantaneous noise estimation allows it to track non-stationary signals effectively. Future work will focus on reducing the computational cost for real-time applications and incorporating temporal continuity constraints to further enhance speech quality.

Summary updated

BibTeX
@inproceedings{raj2010non,
  author = {Bhiksha Raj and Tuomas Virtanen and Sourish Chaudhuri and Rita Singh},
  title = {{Non-negative matrix factorization based compensation of music for automatic speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-268},
}
Figure