Dementia Detection by Fusing Speech and Eye-Tracking Representation
Lists
Only you see this
Abstract
This paper proposes a method of detecting dementia from the simultaneous speech and eye-tracking recordings of subjects in a picture description task. First, automatic speech recognition (ASR) and regional picture recognition (RPR) models are built to extract content-related bottleneck (BN) features for both speech and eye-tracking inputs. Then, a neural network is designed to fuse these two modals for discriminating dementia patients from healthy controls. The network contains a cross-modal Transformer encoder for bimodal interaction, and a self-attention Transformer encoder for final classification. Experimental results demonstrate that the detection accuracy of the proposed method is 84.26%, which outperforms baseline methods and ablated models using single speech or eye-tracking input.
BibTeX
@inproceedings{sheng2022dementia,
author = {Zhengyan Sheng and Zhiqiang Guo and Xin Li and Yunxia Li and Zhenhua Ling},
title = {{Dementia Detection by Fusing Speech and Eye-Tracking Representation}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2022},
url = {https://openreview.net/forum?id=H5fJBTLilQ},
}Loading related content...