Probabilistic features for connecting eye gaze to spoken language understanding

Anna Prokofieva, Malcolm Slaney, Dilek Hakkani-TürICASSP 2015 · OpenReview

Abstract

Many users obtain content from a screen and want to make requests of a system based on items that they have seen. Eye-gaze information is a valuable signal in speech recognition and spoken-language understanding (SLU) because it provides context for a user's next utterance-what the user says next is probably conditioned on what they have seen. This paper investigates three types of features for connecting eye-gaze information to an SLU system: lexical, and two types of eye-gaze features. These features help us to understand which object (i.e. a link) that a user is referring to on a screen. We show a 17% absolute performance improvement in the referenced-object F-score by adding eye-gaze features to conventional methods based on a lexical comparison of the spoken utterance and the text on the screen.
BibTeX
@inproceedings{prokofieva2015probabilistic,
  author = {Anna Prokofieva and Malcolm Slaney and Dilek Hakkani-Tür},
  title = {{Probabilistic features for connecting eye gaze to spoken language understanding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=2C-VhzPsRBv},
}

Loading related content...

Figure