Deep multimodal learning for Audio-Visual Speech Recognition

Youssef Mroueh, Etienne Marcheret, Vaibhava Goel

TL;DR

This paper introduces novel deep learning techniques for Audio-Visual Automatic Speech Recognition (AV-ASR), demonstrating that visual cues significantly improve speech recognition even in clean audio conditions. They achieve state-of-the-art results by fusing unimodal networks and introducing a novel bilinear softmax layer to better capture cross-modal correlations.

What Problem Does This Paper Address?

Human speech perception is inherently multimodal, relying on both auditory and visual information (lip movements, facial expressions). While this is evident in phenomena like the McGurk effect, traditional Automatic Speech Recognition (ASR) systems primarily focus on audio. Audio-Visual Automatic Speech Recognition (AV-ASR) aims to leverage both audio and video streams to improve recognition accuracy, especially in noisy environments where visual cues become crucial for disambiguation. However, this paper specifically investigates the benefit of visual information in clean audio scenarios, challenging the notion that visual cues are only useful when audio is degraded. The core problem is how to effectively fuse these two modalities within a deep learning framework to achieve superior speech recognition performance.

Methods

The authors propose two main deep learning approaches for AV-ASR:

  1. Bimodal DNN Fusion: This method involves training separate Deep Neural Networks (DNNs) for the audio and visual modalities independently. The final hidden layers of these unimodal networks are then concatenated to create a joint feature representation. A new DNN (or a simple softmax layer) is then trained on this fused feature space. This approach aims to learn a shared representation by combining the strengths of individually trained models.
  2. Bilinear Bimodal DNN: To better capture complex correlations between audio and visual features, a novel bilinear softmax layer is introduced. This architecture allows for joint training of the audio and visual networks, directly modeling the interactions between their intermediate features. The paper further proposes a "factored bilinear softmax" to reduce computational complexity, especially for a large number of classes, by sharing weights across related phoneme states. This is achieved by organizing phoneme states in a tree structure and sharing bilinear terms at parent nodes.

Datasets

The primary dataset used for training and evaluation is the IBM AV-ASR Large Vocabulary Studio Dataset. This dataset comprises 40 hours of audio-visual recordings from 262 speakers under clean, studio conditions. The audio is sampled at 16 KHz, and video is captured at 30 frames per second with a resolution of 704x480. The vocabulary size is 10,400 words. A test set of 2 hours from 22 speakers was used for evaluation.

Key Insights and Ideas

  • Visual Cues Matter Even in Clean Audio: A significant finding is that visual information provides substantial benefits for speech recognition even when the audio is clear and free of noise. This challenges the common assumption that visual cues are primarily a fallback for noisy conditions.
  • Bilinear Modeling Captures Deeper Correlations: The introduction of a bilinear softmax layer allows for a more sophisticated modeling of inter-modal correlations compared to simple concatenation. This leads to further performance improvements.
  • Factored Bilinear for Efficiency: For large-scale problems like phoneme classification with many states, a factored bilinear approach with weight sharing significantly reduces computational complexity while maintaining performance. This is crucial for practical deployment.
  • Uncorrelated Errors: Combining posteriors from the bimodal (concatenation) and bilinear models yields further gains, suggesting that these different fusion strategies capture complementary information and have uncorrelated errors.

Key Results

  • The unimodal audio-only DNN achieved a Phone Error Rate (PER) of 41.25%.
  • The unimodal visual-only DNN achieved a PER of 69.36%.
  • The bimodal DNN with fused hidden layers (concatenation) achieved a PER of 35.83%, demonstrating a significant gain from visual information.
  • A shallow fusion (softmax only) on the fused features yielded a similar PER of 35.77%.
  • The bilinear bimodal DNN, trained jointly, further reduced the PER to 34.03% when combined with the fused model.

Limitations and Future Work

  • Computational Complexity: While the factored bilinear approach addresses complexity, the bilinear models are inherently more computationally intensive than simple concatenation.
  • Dataset Specificity: The experiments are conducted on a specific clean audio-visual dataset. The performance on noisier or more diverse datasets might differ.
  • Future Work: The paper suggests exploring more complex architectures and potentially applying these multimodal learning techniques to other related tasks. The success of combining posteriors from different models also points towards ensemble methods as a promising direction.

Summary updated

Abstract

In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of 41% under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of 35.83% demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of 34.03%.
BibTeX
@inproceedings{mroueh2015deep,
  author = {Youssef Mroueh and Etienne Marcheret and Vaibhava Goel},
  title = {{Deep multimodal learning for Audio-Visual Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=MVZ8clpseDD},
}
Figure