Learning Edit Machines for Robust Multimodal Understanding

Michael Johnston, Srinivas BangaloreICASSP 2006 · OpenReview

Abstract

Multimodal grammars provide an expressive formalism for multimodal integration and understanding. However, hand-crafted multimodal grammars can be brittle with respect to unexpected, erroneous, or disfluent inputs. In previous work, we have shown how the robustness of stochastic language models can be combined with the expressiveness of multimodal grammars by adding a finite-state edit machine to the multimodal language processing cascade. In this paper, we present an approach where the edits are trained from data using a noisy channel model paradigm. We evaluate this model and compare its performance against hand-crafted edit machines from our previous work in the context of a multimodal conversational system (MATCH)
BibTeX
@inproceedings{johnston2006learning,
  author = {Michael Johnston and Srinivas Bangalore},
  title = {{Learning Edit Machines for Robust Multimodal Understanding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2006},
  url = {https://openreview.net/forum?id=G1txSdre1Ti},
}

Loading related content...

Figure