A factorial HMM aproach to robust isolated digit recognition in background music

Mark Hasegawa-Johnson, Ameya N. DeorasInterspeech 2004 · DOI

Title: A Factorial HMM Approach to Robust Isolated Digit Recognition in Background Music
Authors: (Not specified in source)
Conference: Interspeech 2004

Abstract

Automatic Speech Recognition (ASR) systems often struggle in environments with non-stationary background noise, such as music. This paper presents a robust recognition approach using Factorial Hidden Markov Models (FHMM) to simultaneously model speech and music signals. By leveraging the MIXMAX approximation—which assumes the log-magnitude spectrum of a mixture is dominated by the louder component—the system effectively separates competing signals. Experimental results on the TIDIGITS corpus mixed with Mozart string quartets demonstrate a 57% relative reduction in word error rate (WER) at 0 dB SNR, significantly outperforming standard HMM baselines.

Introduction

Traditional ASR systems are highly sensitive to environmental noise. While stationary noises (e.g., white noise or steady hums) can be mitigated using spectral subtraction or Wiener filtering, non-stationary signals like music present a unique challenge. Music possesses its own temporal structure and "grammar," often appearing to the recognizer as a competing speaker. At a signal-to-noise ratio (SNR) of 0 dB, where speech and music are equally loud, standard HMM-based recognizers typically experience Word Error Rates (WER) exceeding 75%.

Methodology

The proposed solution utilizes a Factorial Hidden Markov Model (FHMM). Unlike a standard HMM that uses a single state sequence, an FHMM employs parallel chains to represent independent signal sources. In this study, one chain models the spoken digits (0-9) while the second chain models the background music.

Structure of the Factorial Hidden Markov Model (FHMM) showing parallel state chains for speech and music.

A critical component of this architecture is the MIXMAX approximation. This principle posits that in the log-spectral domain, the sum of two signals is approximately equal to the element-wise maximum of their individual spectra. This allows the system to calculate the probability of a mixed observation based on the individual pre-trained models for speech and music without requiring joint training on mixed data.

To maintain computational tractability, the two-chain FHMM is transformed into a single "equivalent" HMM. If the speech model contains $Q$ states and the music model contains $R$ states, the resulting equivalent model spans $Q \times R$ states. The system architecture, including the extension to Gaussian Mixture Models (GMMs), is illustrated below.

System flowchart illustrating the integration of isolated digit and music HMMs into an equivalent HMM.

Experimental Setup

The system was evaluated using the TIDIGITS corpus for speech and Mozart’s string quartets for background music. Training was conducted on clean signals: 50 male speakers for digits and the "Allegro Molto" movement for music. Testing involved different speakers and separate musical movements ("Andante" and "Allegro Assai") to ensure the model's ability to generalize beyond specific training samples. Features were extracted as 20 Mel Frequency Spectral Coefficients (MFSCs).

Results and Discussion

The FHMM approach demonstrated remarkable resilience to high-intensity background music. At 0 dB SNR, the baseline HMM system reached a WER of approximately 76%. In contrast, the FHMM system reduced the WER to roughly 34-35%, representing a 57% relative improvement. This performance remained consistent even across different musical textures; for instance, in the faster "Allegro Assai" test, the FHMM maintained its error rate while the baseline plummeted to 83% WER.

Despite these gains, the FHMM approach faces challenges regarding scalability. The state-space explosion ($Q \times R$) increases computational overhead, making it difficult to apply to large-vocabulary continuous speech recognition. Furthermore, the current music model is specialized for string quartets, and its performance on other genres remains a subject for future investigation.

Conclusion

By treating background music as a structured signal rather than random noise, the FHMM framework provides a powerful tool for robust ASR. The combination of parallel state modeling and the MIXMAX approximation allows for effective speech recognition in highly adverse acoustic environments by effectively "decoding" both the music and the speech simultaneously.

Summary updated

BibTeX
@inproceedings{hasegawajohnson2004factorial,
  author = {Mark Hasegawa-Johnson and Ameya N. Deoras},
  title = {{A factorial HMM aproach to robust isolated digit recognition in background music}},
  booktitle = {Interspeech},
  year = {2004},
  doi = {10.21437/interspeech.2004-637},
}
Figure