Speech Paralinguistic Approach for Detecting Dementia Using Gated Convolutional Neural Network

Mariana Rodrigues Makiuchi, Tifani Warnita, Nakamasa Inoue, Koichi Shinoda, Michitaka Yoshimura, Momoko Kitazawa, Kei Funaki, Yoko Eguchi, Taishiro Kishimoto

TL;DR

This paper presents a non-invasive, audio-only approach for detecting dementia by applying Gated Convolutional Neural Networks (GCNNs) to paralinguistic speech features. By focusing on how people speak rather than what they say, the model achieves up to 80.8% accuracy and can provide a preliminary screening using as little as 4 seconds of audio, offering a cost-effective alternative to traditional transcript-based methods.

What Problem Does This Paper Address?

Dementia diagnosis is traditionally a grueling process involving invasive tests, expensive brain imaging (MRI/PET), and lengthy clinical assessments. While recent research has turned to machine learning for automated detection, most successful models rely on linguistic features—the specific words and syntax used by patients.

The catch? Linguistic analysis requires text transcripts. Generating these manually is expensive and slow, while Automatic Speech Recognition (ASR) often fails on the degraded or slurred speech common in dementia patients. The authors address this gap by proposing a system that relies solely on paralinguistic features (acoustic cues like pitch, jitter, and rhythm) extracted directly from raw audio, eliminating the need for error-prone transcriptions.

Methods

The researchers utilize a combination of signal processing and deep learning to capture the temporal nuances of speech:

  1. Feature Extraction: They use the IS10 feature set (from the OpenSMILE toolkit), which consists of 76 Low-Level Descriptors (LLDs) per time frame, including Mel-frequency cepstral coefficients (MFCCs), pitch, and voicing probability.
  2. Gated Convolutional Neural Networks (GCNN): To model how these acoustic features change over time, the authors employ a GCNN architecture. Unlike standard CNNs, GCNNs use a "gating mechanism" to control the flow of information through the network.
  3. 1D Convolutions: The kernels slide only along the time axis (as seen in Figure 1), treating the speech as a time-series signal.
  4. Gating Mechanism: As illustrated in Figure 2, the input passes through two parallel paths. One path applies a linear transformation, while the other applies a sigmoid activation to act as a "gate." The two are combined via a Hadamard product, which helps the network learn complex temporal patterns while mitigating the "vanishing gradient" problem common in deep networks.
A convolution layer over LLD features
Figure 1: The 1D convolution approach used to process acoustic features over time.
A GCNN with one gated block
Figure 2: The architecture of a single gated block within the GCNN.

Datasets

The model was evaluated on two distinct datasets to test its robustness across languages:

  • DementiaBank Pitt Corpus: A standard English-language dataset containing speech from the "Cookie Theft" picture description task.
  • PROMPT Database: A Japanese-language dataset collected by the authors at Keio University. This dataset is particularly valuable as it includes patients with Mild Cognitive Impairment (MCI), the transitional stage between normal aging and dementia.

Key Insights and Ideas

  • Speed vs. Accuracy Trade-off: One of the most significant findings is that dementia cues are present even in very short bursts of speech. The model achieved ~75% accuracy using just 4 seconds of audio. This suggests that paralinguistic "micro-signatures" are highly discriminative.
  • Language Independence: The method performed well on both English and Japanese datasets. Because it ignores linguistic content, it is theoretically easier to adapt to different languages or low-resource dialects where high-quality ASR doesn't exist.
  • The MCI Challenge: Classifying Mild Cognitive Impairment remains difficult. The authors found that MCI speech patterns are subtle and don't always sit neatly between "Healthy" and "Dementia" in a linear way, suggesting that MCI might require its own specialized acoustic modeling.

Key Results

The GCNN approach consistently outperformed traditional baselines like Support Vector Machines (SVM).

  • Pitt Corpus (English): Achieved 73.1% accuracy, significantly higher than the 67.5% achieved by the SVM baseline.
  • PROMPT Database (Japanese): Achieved a session-level accuracy of 80.8% when using all available speech data.
  • Short-Duration Performance: On the Japanese dataset, the model maintained 74.7% accuracy with only 4 seconds of speech and 77.1% with 20 seconds.
Accuracy vs Speech segment length
Figure 3: Accuracy improves as the speech segment length increases, peaking around 4 seconds for segment-level classification on the Pitt Corpus.
Accuracy vs Speech duration
Figure 4: The relationship between total speech duration used and final accuracy on the PROMPT database.

Limitations and Future Work

While promising, the audio-only approach still lags behind state-of-the-art linguistic methods (which can reach >90% accuracy on the Pitt Corpus). The authors acknowledge that their method is intended as a fast, low-cost screening tool rather than a definitive diagnostic replacement.

Future research directions include:

  • Multimodal Fusion: Combining speech audio with facial expressions and body motion data to improve detection.
  • MCI Refinement: Deepening the analysis of MCI speech to better distinguish it from early-stage dementia and normal aging.
  • Real-world Deployment: Testing the model in noisier, non-clinical environments to see if the paralinguistic cues remain robust.

Summary updated

Abstract

We propose a non-invasive and cost-effective method to automatically detect dementia by utilizing solely speech audio data. We extract paralinguistic features for a short speech segment and use Gated Convolutional Neural Networks (GCNN) to classify it into dementia or healthy. We evaluate our method on the Pitt Corpus and on our own dataset, the PROMPT Database. Our method yields the accuracy of 73.1% on the Pitt Corpus using an average of 114 seconds of speech data. In the PROMPT Database, our method yields the accuracy of 74.7% using 4 seconds of speech data and it improves to 80.8% when we use all the patient's speech data. Furthermore, we evaluate our method on a three-class classification problem in which we included the Mild Cognitive Impairment (MCI) class and achieved the accuracy of 60.6% with 40 seconds of speech data.
BibTeX
@article{makiuchi2021speech,
  author = {Mariana Rodrigues Makiuchi and Tifani Warnita and Nakamasa Inoue and Koichi Shinoda and Michitaka Yoshimura and Momoko Kitazawa and Kei Funaki and Yoko Eguchi and Taishiro Kishimoto},
  title = {{Speech Paralinguistic Approach for Detecting Dementia Using Gated Convolutional Neural Network}},
  journal = {IEICE Trans. Information and Systems},
  year = {2021},
  doi = {10.1587/transinf.2020edp7196},
}
Figure