Non-audible murmur recognition based on fusion of audio and visual streams

Panikos Heracleous, Norihiro HagitaInterspeech 2010

TL;DR

This paper tackles the challenge of recognizing "non-audible murmurs" (NAMs) – extremely soft speech that's inaudible to others – by fusing audio captured by a special ear-worn microphone with visual lip movements. They demonstrate that combining these two streams significantly improves recognition accuracy, especially in noisy environments, by overcoming the inherent spectral limitations of NAM audio.

What Problem Does This Paper Address?

Recognizing speech that is intentionally whispered or spoken so softly it's inaudible to anyone nearby presents a significant challenge. This "non-audible murmur" (NAM) speech, captured by specialized microphones placed behind the ear, suffers from severe low-pass filtering due to body transmission and lack of lip radiation. This spectral reduction, combined with the unvoiced nature of NAM, leads to high confusion rates between similar-sounding phonemes. Traditional audio-only recognition systems struggle with this. The authors aim to improve NAM recognition by integrating visual information from lip movements, hypothesizing that this multimodal approach can compensate for the audio signal's deficiencies.

Methods

The core of this work lies in fusing audio and visual information for NAM speech recognition. They explore three fusion strategies:

  • Concatenative Feature Fusion: This is the simplest method, directly concatenating the audio NAM feature vectors (36 dimensions: MFCC, AMFCC, △△MFCC) with visual feature vectors (15 dimensions derived from lip movements) into a single, larger feature vector (51 dimensions). This is a "state synchronous" method, meaning audio and visual features are aligned in time.
  • Multi-stream HMM Decision Fusion: This is a more sophisticated "state synchronous" approach. It uses separate Hidden Markov Models (HMMs) for the audio and visual streams. The emission probabilities of these individual HMMs are then combined, weighted by stream-specific weights (λn and λv), to form a multi-stream HMM. This method aims to leverage the reliability of each stream.
  • Late Fusion: This method allows for asynchrony between the audio and visual streams. Two independent HMM classifiers are trained, one for NAM audio and one for visual data. For each test utterance, the likelihood scores from both classifiers are combined using a weighted sum (log-likelihoods) to produce a final bi-modal hypothesis. This is particularly useful as audio and visual cues might not always perfectly align in time.

Visual features are extracted using Principal Component Analysis (PCA) on 3D lip point data, capturing articulatory parameters. These video parameters are then interpolated to synchronize with the audio frame rate.

Datasets

The experiments were conducted using a corpus of 212 continuous Japanese utterances, containing 7518 phoneme realizations. For training, 5132 phonemes were used to train 43 monophones. For testing, 2386 phonemes were used. The paper does not mention the introduction of a new benchmark dataset; rather, it utilizes an existing corpus for evaluation.

Key Insights and Ideas

  • Visual Cues are Crucial for Distinguishing Similar Sounds: The paper strongly suggests that while NAM audio is spectrally limited, visual lip movements provide critical disambiguating information, especially for consonants.
  • Late Fusion Outperforms Other Methods: The "late fusion" approach, which allows for temporal asynchrony between audio and visual streams, yielded the best classification accuracies. This highlights that perfect temporal alignment isn't always necessary and can sometimes be a hindrance.
  • NAM Audio is Inherently Confusing for Consonants: The unvoiced nature of NAM and the spectral filtering mean that phonemes articulated in similar places become acoustically very alike, leading to high confusion rates for consonants.
  • Noisy Environments Benefit Most from Fusion: The integration of visual information provided a substantial improvement in accuracy even in noisy conditions, demonstrating the robustness of the multimodal approach.

Key Results

  • Clean Environment:
    • Late fusion achieved the highest phoneme classification accuracy at 71.8%, compared to multi-stream HMM fusion (68.9%) and feature fusion (67.8%).
  • For vowels, late fusion reached 86.2% accuracy, and for consonants, 64.1%.
  • When fusing NAM audio with visual data, a relative improvement of 30% for phoneme recognition was observed compared to using NAM audio alone.
  • Noisy Environment:
    • Integrating visual information with NAM speech resulted in a significant improvement, with an average of 15% absolute increase in accuracy across different Signal-to-Noise Ratio (SNR) levels.
  • Statistical Significance: McNemar's tests confirmed that the observed improvements in accuracy when using audio-visual fusion were statistically significant (p-values of 0.0001).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the unvoiced nature of NAM and spectral reduction inherently lead to higher confusion rates for consonants compared to vowels.
  • The visual parameter extraction relies on specific markers and good lighting conditions, which might limit real-world applicability without further robustness improvements.
  • Future Work:
    • The authors suggest that further investigation into the optimal weighting of audio and visual streams in fusion methods could lead to even better performance.
  • Exploring more advanced fusion techniques or different visual feature extraction methods could also be beneficial.
  • The paper implicitly suggests extending this work to noisy environments, which they do explore, but further refinement in this area is always a possibility.

Summary updated

BibTeX
@inproceedings{heracleous2010non,
  author = {Panikos Heracleous and Norihiro Hagita},
  title = {{Non-audible murmur recognition based on fusion of audio and visual streams}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure