Dementia Detection by Fusing Speech and Eye-Tracking Representation

Zhengyan Sheng, Zhiqiang Guo, Xin Li, Yunxia Li, Zhenhua LingICASSP 2022 · OpenReview

Abstract

This paper proposes a method of detecting dementia from the simultaneous speech and eye-tracking recordings of subjects in a picture description task. First, automatic speech recognition (ASR) and regional picture recognition (RPR) models are built to extract content-related bottleneck (BN) features for both speech and eye-tracking inputs. Then, a neural network is designed to fuse these two modals for discriminating dementia patients from healthy controls. The network contains a cross-modal Transformer encoder for bimodal interaction, and a self-attention Transformer encoder for final classification. Experimental results demonstrate that the detection accuracy of the proposed method is 84.26%, which outperforms baseline methods and ablated models using single speech or eye-tracking input.
BibTeX
@inproceedings{sheng2022dementia,
  author = {Zhengyan Sheng and Zhiqiang Guo and Xin Li and Yunxia Li and Zhenhua Ling},
  title = {{Dementia Detection by Fusing Speech and Eye-Tracking Representation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=H5fJBTLilQ},
}

Loading related content...

Figure