Design, construction and evaluation of emotional multimodal pathological speech database

Ting Zhu, Shufei Duan, Huizhi Liang, Wei ZhangarXiv 2023 · DOI · arXiv

Abstract

Dysarthria, a motor speech disorder often resulting from neurological damage such as stroke, significantly impacts a patient's quality of life. While mechanical speech impairments are well-documented, the emotional dimensions of pathological speech remain under-researched, particularly in Mandarin Chinese. This paper introduces THE-POSSD, the first Chinese multimodal emotional pathological speech database. By integrating audio, video, glottal signals, and brain activity (fNIRS), the study demonstrates that motor dysarthria not only impairs speech intelligibility but also significantly degrades emotional expression, correlating strongly with psychological distress.

Introduction

Patients with dysarthria frequently experience affective disorders like depression and anxiety. However, most existing speech databases are "emotion-neutral" and limited to single modalities. THE-POSSD (Multimodal Emotional Database of Post-Stroke Speech Dysfunction Patients) addresses this gap by providing a comprehensive dataset to study the intersection of motor impairment, emotional expression, and psychological health in Mandarin speakers.

Methods and Data Collection

The researchers constructed THE-POSSD using a rigorous multimodal pipeline involving 68 participants (39 post-stroke patients and 29 healthy controls).

The experimental setup for multimodal data collection, featuring synchronized recording of acoustic, glottal, and physiological signals.

Multimodal Capture

Data collection included:

  • Acoustic & Glottal Signals: High-quality audio paired with Electroglottograph (EGG) data.
  • Physiological & Brain Activity: 22-channel functional near-infrared spectroscopy (fNIRS) for brain oxygenation and pulse oximetry for heart rate.
  • Visual Data: High-definition video for facial expression analysis.

Emotion Induction

To elicit genuine emotions (Happy, Sad, Angry, Neutral), the study utilized video clips and "Stanislavsky’s emotional memory approach," where subjects recalled personal experiences to trigger specific affective states.

Key Insights: The Emotional Toll of Dysarthria

Psychological Correlation

Using the Symptom Checklist-90 (SCL-90), the study found that psychological factors—including depression, anxiety, and interpersonal sensitivity—are significantly higher in dysarthria patients compared to healthy controls. These psychological symptoms scale directly with the severity of the motor impairment.

Mean scores of psychological factors (SCL-90) across different dysarthria severity levels, showing a clear correlation between physical impairment and psychological distress.

Emotional "Flattening" and Confusion

The study confirms that as physical speech ability declines, the ability to convey emotion through voice degrades, leading to a "flattening" of expression. Interestingly, in the Valence-Arousal (VA) spatial distribution, "happy" expressions in patients often crossed into the "angry" quadrant. This occurs because patients frequently use a strained or "excited" voice to compensate for motor limitations, which can confuse both human listeners and automated algorithms.

Evaluation and Results

Human vs. Machine Recognition

  • Subjective Accuracy: Human annotators achieved high emotion discrimination (above 87.5%), with neutral emotions being the easiest to identify.
  • Speech Intelligibility: While healthy controls were rated at 93.8% intelligibility, this dropped to 67.3% for severe dysarthria patients.
  • Automated Recognition: Machine learning models (SVM and Random Forest) achieved 78% accuracy for controls but only 60% for patients using audio data. Glottal data proved even more challenging for emotion classification, though it remained highly effective for classifying disease severity (up to 87% accuracy).

Conclusion and Future Work

THE-POSSD provides a unique resource for the study of pathological speech. While acoustic data remains superior for emotion recognition, glottal and physiological data offer critical insights into the physical impact of the disease. Future research will focus on the fusion of video and fNIRS data to create a more robust, multidimensional model of pathological speech and its emotional components.

Summary updated

Abstract

The lack of an available emotion pathology database is one of the key obstacles in studying the emotion expression status of patients with dysarthria. The first Chinese multimodal emotional pathological speech database containing multi-perspective information is constructed in this paper. It includes 29 controls and 39 patients with different degrees of motor dysarthria, expressing happy, sad, angry and neutral emotions. All emotional speech was labeled for intelligibility, types and discrete dimensional emotions by developed WeChat mini-program. The subjective analysis justifies from emotion discrimination accuracy, speech intelligibility, valence-arousal spatial distribution, and correlation between SCL-90 and disease severity. The automatic recognition tested on speech and glottal data, with average accuracy of 78% for controls and 60% for patients in audio, while 51% for controls and 38% for patients in glottal data, indicating an influence of the disease on emotional expression.
BibTeX
@misc{zhu2023design,
  author = {Ting Zhu and Shufei Duan and Huizhi Liang and Wei Zhang},
  title = {{Design, construction and evaluation of emotional multimodal pathological speech database}},
  year = {2023},
  eprint = {2312.08998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.08998},
}
Figure