Learning Co-Speech Gesture for Multimodal Aphasia Type Detection

Daeun Lee, Sejung Son, Hyolim Jeon, Seungbae Kim, Jinyoung HanEMNLP 2023 · DOI

Introduction

Aphasia is a complex language disorder typically resulting from brain damage, such as a stroke, which significantly impairs communication. While clinicians can easily identify the presence of aphasia, diagnosing specific types—such as Broca’s (non-fluent) versus Wernicke’s (fluent but incoherent)—is a resource-intensive process requiring expert evaluation. Current automated diagnostic tools often rely solely on linguistic or acoustic data, overlooking the critical role of non-verbal communication. People with aphasia (PWA) frequently use hand gestures to compensate for word-retrieval difficulties, making these co-speech gestures a vital clinical marker.

Methodology

The researchers proposed a multimodal graph neural network architecture designed to fuse speech and gesture data effectively. The system consists of four primary modules:

  1. Speech-Gesture Graph Encoder: Constructs a heterogeneous graph where nodes represent disfluency keywords, audio features, and gesture tokens. A GraphSAGE-based encoder learns the intricate correlations between these modalities.
  2. Gesture-aware Word Embedding Layer: Refines pre-trained RoBERTa embeddings using gesture-informed representations, making textual features sensitive to physical motion.
  3. Multimodal Fusion Encoder: Utilizes a Multimodal Transformer (MulT) architecture with cross-modal attention to integrate refined text, audio (OpenSmile), and visual (MediaPipe) features.
  4. Aphasia Type Prediction Decoder: A final classification network that categorizes input into Control, Fluent, Non-Comprehension, or Non-Fluent types.

Dataset and Preprocessing

The study utilized the AphasiaBank clinical database, focusing on the Cinderella Story Recall task. The dataset included 3,651 aligned multimodal samples from 507 subjects.

  • Transcription: Whisper ASR was used to capture fillers and disfluencies often omitted by standard ASR.
  • Gesture Extraction: MediaPipe extracted 3D coordinates for 23 upper-body keypoints.
  • Categorization: Samples were labeled into four clinical categories: Control, Fluent (Anomic, Conduction), Non-Comprehension (Sensory, Wernicke), and Non-Fluent (Trans. Motor, Broca).

Key Insights and Results

The proposed model achieved a state-of-the-art weighted average F1 score of 84.2%, significantly outperforming baseline methods like Random Forests (74.2%) and other multimodal fusion techniques like MISA (76.1%).

Multimodal representations of different aphasia types, showing the alignment of visual (V), textual (T), and acoustic (A) tokens.

A central finding of the research is that visual gesture features are more predictive of aphasia types than acoustic features. The model demonstrates that the way a patient moves provides a more distinct clinical marker than the sound of their voice. Furthermore, the study highlights the importance of disfluencies; rather than being "noise," stutters and filler words (like "um" or "[*]") serve as critical anchor points where meaningful speech-gesture correlations occur. For instance, individuals with Broca’s aphasia often use iconic gestures to preserve semantic meaning despite impaired verbal fluency.

Limitations and Future Directions

Despite its success, the model faces challenges related to data scarcity for specific subtypes (e.g., Sensory aphasia) and the inherent overlap in traditional clinical classification schemes. Future work could benefit from incorporating mouth landmark data—which was unavailable in this study due to camera distance—and collaborating with speech-language pathologists to refine the selection of disfluency keywords.

Summary updated

BibTeX
@inproceedings{lee2023learning,
  author = {Daeun Lee and Sejung Son and Hyolim Jeon and Seungbae Kim and Jinyoung Han},
  title = {{Learning Co-Speech Gesture for Multimodal Aphasia Type Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.577},
}
Figure