Analysis of correlation between audio and visual speech features for clean audio feature prediction in noise

Ibrahim Almajai, Ben Milner, Jonathan DarchInterspeech 2006

TL;DR

This paper investigates the correlation between audio and visual speech features to predict clean audio features from visual cues, aiming to improve speech enhancement in noisy environments. They found that broader spectral audio features correlate better with visual features, and successfully predicted formant frequencies from visual data, paving the way for noise-robust speech processing.

What Problem Does This Paper Address?

Humans naturally use visual cues from lip movements to understand speech, especially when the audio is noisy. This phenomenon, known as audio-visual speech perception, has led to research in combining audio and visual information for speech recognition and enhancement. The core challenge this paper addresses is how to effectively leverage visual speech information to improve the quality of noisy audio signals. Specifically, they aim to find visual features that can accurately predict clean audio features, which can then be used to reconstruct or enhance the degraded audio. This is crucial because directly processing noisy audio can lead to significant degradation, and relying solely on visual information for audio reconstruction is a promising avenue for robust speech enhancement.

Methods

The authors explore the correlation between audio and visual speech features using multiple linear regression. They extract two types of audio features: Mel-Frequency Cepstral Coefficients (MFCCs) and formant frequencies. For visual features, they employ three methods: an Active Appearance Model (AAM), 2D Discrete Cosine Transform (DCT), and Cross-DCT.

  • Audio Feature Extraction:
    • MFCCs: Standard MFCCs (0-12) and log energy are extracted, forming a 14-dimensional vector.
  • Formant Frequencies: The first four formant frequencies (F1-F4) are extracted using linear predictive analysis and refined with Kalman filtering, resulting in a 4-dimensional vector.
  • Visual Feature Extraction:
    • AAM: A statistical model that captures both shape and appearance of facial features, specifically focusing on the lower half of the face.
  • 2D DCT: Applied to a region of interest around the mouth, concentrating image energy into lower coefficients.
  • Cross DCT: A simpler approach involving 1D DCTs on horizontal and vertical pixel vectors from the mouth's center.
  • Correlation Measurement: Multiple linear regression is used to model the relationship between visual features (independent variables) and audio features (dependent variables). The R-squared term quantifies the correlation.
  • Audio Feature Prediction: A Gaussian Mixture Model (GMM) is trained to model the joint distribution of audio and visual features. This GMM is then used to predict audio features from visual features using a Maximum A Posteriori (MAP) estimation.

Datasets

The experiments were conducted on an audio-visual speech database comprising 277 sentences of continuous speech spoken by a single male speaker. Of these, 200 utterances were used for training and 77 for testing. The audio data was sampled at 8kHz, and the video was originally recorded at 25 frames per second, upsampled to 100 frames per second to match the audio frame rate.

Key Insights and Ideas

  • Broad Spectral Features are Key: Audio features that capture broader spectral information (like MFCCs 0-4 and log energy) show higher correlation with visual features compared to those representing finer spectral details (higher-order MFCCs). This suggests that the overall shape and energy of the speech signal are more readily discernible from lip movements.
  • Visual Feature Independence: Interestingly, the three different visual feature extraction methods (AAM, 2D DCT, Cross-DCT) exhibited very similar levels of correlation with audio features, despite their distinct computational approaches. This implies that the information contained in facial movements relevant to speech is robust across different feature extraction strategies.
  • Formant Prediction Success: The study demonstrated that formant frequencies, particularly F1 and F2, can be predicted with reasonable accuracy from visual features. This is significant because formants are direct indicators of vocal tract shape, and their accurate prediction from visual cues opens doors for reconstructing spectral envelope information.
  • GMM for Prediction: The use of a GMM proved effective in modeling the complex joint distribution of audio and visual features, enabling accurate prediction of audio features from visual input.

Key Results

  • MFCC Correlation: MFCC 0 and log energy showed the highest correlation with visual features, with R-values around 0.85 and 0.82 respectively (with AAM features). Higher-order MFCCs (10-12) had significantly lower correlations.
  • Formant Correlation: The correlation between visual features and formants was lower than with MFCCs. F1 and F2 had the highest correlation (around R=0.56), while F3 and F4 showed progressively lower correlations.
  • Formant Prediction Accuracy: When predicting formant frequencies using the GMM with AAM features, the accuracy increased with the number of clusters up to 16. Figure 4 shows percentage formant frequency error decreasing as the number of clusters increases. Figure 5 visually demonstrates that the predicted F1 contour closely follows the reference F1 contour.

Limitations and Future Work

  • Limitations: The study was conducted on a single male speaker, which might limit the generalizability of the findings to different speakers, genders, or accents. The correlation for higher-order formants (F3, F4) was considerably weaker, and even difficult to discern in spectrograms, suggesting limitations in extracting this fine-grained information visually.
  • Future Work: The authors suggest that the visually predicted clean spectral envelope information can be used for speech enhancement in noisy conditions. This is identified as the subject of further work, implying that the current paper lays the groundwork for developing such systems.

Summary updated

BibTeX
@inproceedings{almajai2006analysis,
  author = {Ibrahim Almajai and Ben Milner and Jonathan Darch},
  title = {{Analysis of correlation between audio and visual speech features for clean audio feature prediction in noise}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure