Introduction
Aphasia is a complex language disorder typically resulting from brain damage, such as a stroke, which significantly impairs communication. While clinicians can easily identify the presence of aphasia, diagnosing specific types—such as Broca’s (non-fluent) versus Wernicke’s (fluent but incoherent)—is a resource-intensive process requiring expert evaluation. Current automated diagnostic tools often rely solely on linguistic or acoustic data, overlooking the critical role of non-verbal communication. People with aphasia (PWA) frequently use hand gestures to compensate for word-retrieval difficulties, making these co-speech gestures a vital clinical marker.
Methodology
The researchers proposed a multimodal graph neural network architecture designed to fuse speech and gesture data effectively. The system consists of four primary modules:
- Speech-Gesture Graph Encoder: Constructs a heterogeneous graph where nodes represent disfluency keywords, audio features, and gesture tokens. A GraphSAGE-based encoder learns the intricate correlations between these modalities.
- Gesture-aware Word Embedding Layer: Refines pre-trained RoBERTa embeddings using gesture-informed representations, making textual features sensitive to physical motion.
- Multimodal Fusion Encoder: Utilizes a Multimodal Transformer (MulT) architecture with cross-modal attention to integrate refined text, audio (OpenSmile), and visual (MediaPipe) features.
- Aphasia Type Prediction Decoder: A final classification network that categorizes input into Control, Fluent, Non-Comprehension, or Non-Fluent types.
Dataset and Preprocessing
The study utilized the AphasiaBank clinical database, focusing on the Cinderella Story Recall task. The dataset included 3,651 aligned multimodal samples from 507 subjects.
- Transcription: Whisper ASR was used to capture fillers and disfluencies often omitted by standard ASR.
- Gesture Extraction: MediaPipe extracted 3D coordinates for 23 upper-body keypoints.
- Categorization: Samples were labeled into four clinical categories: Control, Fluent (Anomic, Conduction), Non-Comprehension (Sensory, Wernicke), and Non-Fluent (Trans. Motor, Broca).
Key Insights and Results
The proposed model achieved a state-of-the-art weighted average F1 score of 84.2%, significantly outperforming baseline methods like Random Forests (74.2%) and other multimodal fusion techniques like MISA (76.1%).
A central finding of the research is that visual gesture features are more predictive of aphasia types than acoustic features. The model demonstrates that the way a patient moves provides a more distinct clinical marker than the sound of their voice. Furthermore, the study highlights the importance of disfluencies; rather than being "noise," stutters and filler words (like "um" or "[*]") serve as critical anchor points where meaningful speech-gesture correlations occur. For instance, individuals with Broca’s aphasia often use iconic gestures to preserve semantic meaning despite impaired verbal fluency.
Limitations and Future Directions
Despite its success, the model faces challenges related to data scarcity for specific subtypes (e.g., Sensory aphasia) and the inherent overlap in traditional clinical classification schemes. Future work could benefit from incorporating mouth landmark data—which was unavailable in this study due to camera distance—and collaborating with speech-language pathologists to refine the selection of disfluency keywords.
Summary updated