Audio-visual deep learning for noise robust speech recognition

Jing Huang, Brian KingsburyICASSP 2013 · OpenReview

Abstract

Deep belief networks (DBN) have shown impressive improvements over Gaussian mixture models for automatic speech recognition. In this work we use DBNs for audio-visual speech recognition; in particular, we use deep learning from audio and visual features for noise robust speech recognition. We test two methods for using DBNs in a multimodal setting: a conventional decision fusion method that combines scores from single-modality DBNs, and a novel feature fusion method that operates on mid-level features learned by the single-modality DBNs. On a continuously spoken digit recognition task, our experiments show that these methods can reduce word error rate by as much as 21% relative over a baseline multi-stream audio-visual GMM/HMM system.
BibTeX
@inproceedings{huang2013audio,
  author = {Jing Huang and Brian Kingsbury},
  title = {{Audio-visual deep learning for noise robust speech recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=hT2B02WQL4O8},
}

Loading related content...

Figure