An Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition

Samy BengioNeurIPS 2002 · OpenReview

Introduction

In multimodal speech recognition, combining audio (microphones) and visual (cameras) data can significantly improve accuracy, especially in noisy environments where audio alone fails. However, audio and video signals are rarely perfectly in sync. For instance, a speaker's lips often start moving before any sound is produced.

Current solutions usually fall into two camps:

  1. Early Integration: Forcing both signals to the same frame rate and modeling them as a single joint vector. This assumes a fixed, a priori alignment that doesn't reflect reality.
  2. Late Integration: Modeling each stream with its own Hidden Markov Model (HMM) and combining the results at the end. This loses the ability to learn the joint relationship between the two modalities during the actual recognition process.

To address this, this paper introduces the Asynchronous Hidden Markov Model (AHMM), a novel architecture designed to model the joint probability of two data streams representing the same event but lacking perfect synchronization. By allowing one stream to "stretch" relative to the other during training and decoding, the AHMM achieves significantly higher accuracy in noisy speech recognition tasks compared to traditional models.

The AHMM Architecture

The AHMM is inspired by Pair HMMs (used in biological sequence alignment) and Asynchronous Input/Output HMMs. Key technical innovations include:

  • Hidden Alignment Variable: The model introduces a hidden variable ($\tau_t$) that tracks the alignment between the sequences. While the state ($q$) is synchronized with the longer sequence (usually audio), the model decides at each step whether or not to emit an observation from the second sequence (usually video).
  • Emission Probability $\epsilon(i, t)$: This represents the probability that the system emits the next observation of the video sequence at time $t$ while in state $i$. This learned parameter allows the model to "stretch" one sequence to find the best match.
  • Algorithmic Adaptation: The standard HMM toolkit is modified to include a Forward procedure for joint likelihood, an Expectation-Maximization (EM) algorithm for training, and a Viterbi decoder to find the most probable path and optimal alignment between audio and video.

To keep the computational complexity manageable, the model constrains alignments to a specific time window (e.g., within 0.5 seconds of each other).

Key Insights: Dynamic Stretching and Alignment

The core innovation of the AHMM is treating alignment as a latent variable to be optimized. It doesn't assume the video frame at time $t$ corresponds exactly to the audio frame at time $t$; instead, it finds the frame that best fits the current state.

Non-linear alignment path between audio and video sequences

As shown by the Viterbi decoder's output, the optimal alignment path between audio and video is highly non-linear. This confirms that the relationship between lip movement and sound is complex and varies throughout a word, justifying the need for an asynchronous model over a rigidly synchronized one. Furthermore, the AHMM effectively "weights" modalities based on reliability. When audio is heavily corrupted, performance gracefully degrades toward the video-only baseline rather than failing completely.

Experimental Results

The model was evaluated using the M2VTS database (French digits 0-9), combining 33 audio features and 48 video features. To test robustness, noise from the Noisex database was injected at various levels (15dB, 10dB, 5dB, and 0dB).

Word Error Rate across different noise levels for various HMM architectures

The AHMM consistently outperformed both standard audio-only HMMs and "naively" synchronized multimodal HMMs in noisy conditions:

  • High Noise (0dB): The audio-only HMM suffered a 79.1% Word Error Rate (WER), and a standard manually synchronized multimodal HMM had a 45.4% WER. The AHMM achieved a 41.1% WER, approaching the video-only performance baseline (39.6%).
  • Moderate Noise (5dB): The AHMM (22.3% WER) significantly outperformed the standard multimodal HMM (35.3% WER) and the audio-only HMM (38.7% WER).
  • Clean Data (15dB): The AHMM maintained high performance (4.8% WER), demonstrating that its added architectural complexity does not penalize accuracy when the audio signal is clear.

Limitations and Future Work

While highly effective, the AHMM's 3D trellis (state $\times$ time $\times$ alignment) is computationally more expensive than standard HMMs, even with alignment window constraints. Additionally, the emission probability $\epsilon(i, t)$ was implemented as a simple table in these preliminary experiments; using more complex distributions conditioned on the input data could yield further improvements. Beyond speech recognition, this architecture holds promise for other asynchronous multimodal tasks, such as biometric person authentication.

Summary updated

Abstract

This paper presents a novel Hidden Markov Model architecture to model the joint probability of pairs of asynchronous sequences de(cid:173) scribing the same event. It is based on two other Markovian models, namely Asynchronous Input/ Output Hidden Markov Models and Pair Hidden Markov Models. An EM algorithm to train the model is presented, as well as a Viterbi decoder that can be used to ob(cid:173) tain the optimal state sequence as well as the alignment between the two sequences. The model has been tested on an audio-visual speech recognition task using the M2VTS database and yielded robust performances under various noise conditions.
BibTeX
@inproceedings{bengio2002asynchronous,
  author = {Samy Bengio},
  title = {{An Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2002},
  url = {https://openreview.net/forum?id=rJWzgOWdWr},
}
Figure