Combining Multiple Views for Visual Speech Recognition

Marina Zimmermann, Mostafa Mehdipour-Ghazi, Hazim Kemal Ekenel, Jean-Philippe Thiran

Visual Speech Recognition (VSR), commonly known as automatic lip-reading, is a challenging task due to the subtle and rapid dynamics of human speech. While traditional research has focused primarily on frontal views, real-world environments—such as security systems or multi-camera smart devices—often provide access to multiple perspectives. This paper investigates whether combining these views can enhance recognition accuracy and explores the optimal strategies for information fusion.

Methodology

The researchers propose a multi-stage "tandem" architecture that blends classical signal processing with modern deep learning:

  1. Feature Extraction (PCANet): A computationally efficient PCA-based convolutional network is used to extract visual features. This architecture is particularly effective when training data is limited.
  2. Temporal Modeling (LSTM): These features are fed into Long Short-Term Memory (LSTM) networks to capture the temporal "flow" of speech.
  3. Sequence Modeling (Tandem HMM): The LSTM outputs are integrated into a Hidden Markov Model (GMM-HMM) to decode the transitions between phonemes and words.
  4. Decision Fusion: This is the core innovation. Instead of merging raw features (feature fusion), the system performs recognition on each view independently. It then combines the resulting "Viterbi path log-likelihoods" (confidence scores) using a weighted average to make a final prediction.

Experimental Results

The system was evaluated on the OuluVS2 dataset, a specialized benchmark featuring 52 subjects recorded from five simultaneous camera angles: 0° (frontal), 30°, 45°, 60°, and 90° (profile).

The "30° Sweet Spot"

A key finding is that the direct frontal view (0°) is not the most informative. The 30° view consistently outperformed others in single-camera tests, likely because the slight angle provides better depth information regarding lip and jaw articulation. While the 90° profile view performs poorly in isolation, it acts as a valuable "helper" view, providing complementary information that frontal perspectives lack.

The Power of Combination

Combining multiple views significantly improves performance. As shown in Figure 1, adding views generally increases recognition correctness across sentence and word metrics.

Recognition performance across different view combinations.

The most effective combination—0°, 30°, and 60°—achieved a sentence correctness of 83.1%, a substantial jump from the 75.6% achieved by the best single view (30°).

Fusion Strategies: Decision vs. Feature

The study demonstrates that decision-level fusion is superior to feature-level fusion. By allowing each camera angle to contribute its own confidence score, the system becomes more robust against noise or poor-quality input from any single angle.

Detailed recognition results comparing fusion strategies and view combinations.

As detailed in Figure 4, decision fusion (Rec) consistently outperforms feature fusion (Feat). However, there is a "limit of more": incorporating all five views without precise weight tuning can introduce noise, leading to a performance drop (e.g., to 75.7% for the five-view combination compared to the 83.3% peak with four views).

Conclusion

This research highlights the significant advantages of multi-view VSR. By leveraging decision fusion and identifying optimal viewing angles, the proposed system achieves high accuracy on standardized benchmarks. Future work will focus on moving toward more complex end-to-end deep learning models and testing these strategies on larger, "in-the-wild" datasets to verify if the 30°/60° advantage holds in unconstrained environments.

Summary updated

Abstract

Visual speech recognition is a challenging research problem with a particular practical application of aiding audio speech recognition in noisy scenarios. Multiple camera setups can be beneficial for the visual speech recognition systems in terms of improved performance and robustness. In this paper, we explore this aspect and provide a comprehensive study on combining multiple views for visual speech recognition. The thorough analysis covers fusion of all possible view angle combinations both at feature level and decision level. The employed visual speech recognition system in this study extracts features through a PCA-based convolutional neural network, followed by an LSTM network. Finally, these features are processed in a tandem system, being fed into a GMM-HMM scheme. The decision fusion acts after this point by combining the Viterbi path log-likelihoods. The results show that the complementary information contained in recordings from different view angles improves the results significantly. For example, the sentence correctness on the test set is increased from 76% for the highest performing single view ($30^\circ$) to up to 83% when combining this view with the frontal and $60^\circ$ view angles.
BibTeX
@inproceedings{zimmermann2017combining,
  author = {Marina Zimmermann and Mostafa Mehdipour-Ghazi and Hazim Kemal Ekenel and Jean-Philippe Thiran},
  title = {{Combining Multiple Views for Visual Speech Recognition}},
  booktitle = {AVSP},
  year = {2017},
  doi = {10.21437/avsp.2017-10},
}
Figure