Introduction
In multimodal speech recognition, combining audio (microphones) and visual (cameras) data can significantly improve accuracy, especially in noisy environments where audio alone fails. However, audio and video signals are rarely perfectly in sync. For instance, a speaker's lips often start moving before any sound is produced.
Current solutions usually fall into two camps:
- Early Integration: Forcing both signals to the same frame rate and modeling them as a single joint vector. This assumes a fixed, a priori alignment that doesn't reflect reality.
- Late Integration: Modeling each stream with its own Hidden Markov Model (HMM) and combining the results at the end. This loses the ability to learn the joint relationship between the two modalities during the actual recognition process.
To address this, this paper introduces the Asynchronous Hidden Markov Model (AHMM), a novel architecture designed to model the joint probability of two data streams representing the same event but lacking perfect synchronization. By allowing one stream to "stretch" relative to the other during training and decoding, the AHMM achieves significantly higher accuracy in noisy speech recognition tasks compared to traditional models.
The AHMM Architecture
The AHMM is inspired by Pair HMMs (used in biological sequence alignment) and Asynchronous Input/Output HMMs. Key technical innovations include:
- Hidden Alignment Variable: The model introduces a hidden variable ($\tau_t$) that tracks the alignment between the sequences. While the state ($q$) is synchronized with the longer sequence (usually audio), the model decides at each step whether or not to emit an observation from the second sequence (usually video).
- Emission Probability $\epsilon(i, t)$: This represents the probability that the system emits the next observation of the video sequence at time $t$ while in state $i$. This learned parameter allows the model to "stretch" one sequence to find the best match.
- Algorithmic Adaptation: The standard HMM toolkit is modified to include a Forward procedure for joint likelihood, an Expectation-Maximization (EM) algorithm for training, and a Viterbi decoder to find the most probable path and optimal alignment between audio and video.
To keep the computational complexity manageable, the model constrains alignments to a specific time window (e.g., within 0.5 seconds of each other).
Key Insights: Dynamic Stretching and Alignment
The core innovation of the AHMM is treating alignment as a latent variable to be optimized. It doesn't assume the video frame at time $t$ corresponds exactly to the audio frame at time $t$; instead, it finds the frame that best fits the current state.
As shown by the Viterbi decoder's output, the optimal alignment path between audio and video is highly non-linear. This confirms that the relationship between lip movement and sound is complex and varies throughout a word, justifying the need for an asynchronous model over a rigidly synchronized one. Furthermore, the AHMM effectively "weights" modalities based on reliability. When audio is heavily corrupted, performance gracefully degrades toward the video-only baseline rather than failing completely.
Experimental Results
The model was evaluated using the M2VTS database (French digits 0-9), combining 33 audio features and 48 video features. To test robustness, noise from the Noisex database was injected at various levels (15dB, 10dB, 5dB, and 0dB).
The AHMM consistently outperformed both standard audio-only HMMs and "naively" synchronized multimodal HMMs in noisy conditions:
- High Noise (0dB): The audio-only HMM suffered a 79.1% Word Error Rate (WER), and a standard manually synchronized multimodal HMM had a 45.4% WER. The AHMM achieved a 41.1% WER, approaching the video-only performance baseline (39.6%).
- Moderate Noise (5dB): The AHMM (22.3% WER) significantly outperformed the standard multimodal HMM (35.3% WER) and the audio-only HMM (38.7% WER).
- Clean Data (15dB): The AHMM maintained high performance (4.8% WER), demonstrating that its added architectural complexity does not penalize accuracy when the audio signal is clear.
Limitations and Future Work
While highly effective, the AHMM's 3D trellis (state $\times$ time $\times$ alignment) is computationally more expensive than standard HMMs, even with alignment window constraints. Additionally, the emission probability $\epsilon(i, t)$ was implemented as a simple table in these preliminary experiments; using more complex distributions conditioned on the input data could yield further improvements. Beyond speech recognition, this architecture holds promise for other asynchronous multimodal tasks, such as biometric person authentication.
Summary updated