Production domain modeling of pronunciation for visual speech recognition

Kate Saenko, Karen Livescu, James R. Glass, Trevor Darrell

Abstract

Articulatory feature models have been proposed in the automatic speech recognition community as an alternative to phone-based models of speech. In this paper, we extend this approach to the visual modality. Specifically, we adapt a recently proposed feature-based model of pronunciation variation to visual speech recognition (VSR) using a set of visually-salient features. The model uses a dynamic Bayesian network (DBN) to represent the evolution of the feature streams. A bank of SVM feature classifiers, with outputs converted to likelihoods, provides input to the DBN. We present preliminary experiments on an isolated-word VSR task, comparing feature-based and viseme-based units and studying the effects of modeling inter-feature asynchrony.
BibTeX
@inproceedings{saenko2005production,
  author = {Kate Saenko and Karen Livescu and James R. Glass and Trevor Darrell},
  title = {{Production domain modeling of pronunciation for visual speech recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2005},
  doi = {10.1109/icassp.2005.1416343},
}

Loading related content...

Figure