Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease

Junan Li, Yunxiang Li, Yuren Wang, Xixin Wu, Helen Meng

Abstract

Alzheimer's disease (AD) has become one of the most significant health challenges in an aging society. The use of spoken language-based AD detection methods has gained prevalence due to their scalability. Based on the Cookie Theft picture description task, we devised an explainable and effective feature set that leverages the visual capabilities of a large language model (LLM) and the Term Frequency-Inverse Document Frequency (TF-IDF) model. Our experimental results show that the newly proposed features consistently outperform traditional linguistic features across two different classifiers with high dimension efficiency. Our new features can be well explained and interpreted step by step which enhance the interpretability of automatic AD screening.

Introduction

Early detection of Alzheimer’s Disease (AD) is critical for timely intervention, yet traditional diagnostic methods are often invasive or prohibitively expensive. Spoken language analysis, particularly using the standardized "Cookie Theft" picture description task, offers a low-cost, non-invasive alternative. However, existing AI-driven approaches often struggle with a trade-off between interpretability and performance. Traditional linguistic features often rely on high-dimensional sets that lack task-specific context, while modern Large Language Models (LLMs) used as direct classifiers act as "black boxes," raising concerns regarding data privacy and computational cost. This research proposes a "middle path": a compact, explainable, and highly effective set of 15 features.

Methods

The researchers developed a novel pipeline that utilizes Multimodal Large Language Models (MLLMs) as tools for feature engineering rather than final decision-makers. The core of this approach involves generating task-specific metrics based on the visual content of the stimulus image.

The pipeline for generating topic-related keywords and reference descriptions using GPT-4o.

The 15-feature set is built on three pillars:

  • Topic-Related Keywords: The "Cookie Theft" image was segmented into three zones (the children, the mother/sink, and the window). GPT-4o generated keyword sets for each, and the "Hit Rate" (presence of these keywords in patient transcripts) was calculated.
  • LLM "Golden Standards": GPT-4o generated 15 "perfect" descriptions of the scene. Patient transcripts were then compared against these using standard machine translation metrics like BLEU and METEOR.
  • TF-IDF Similarity: Cosine similarity was measured between patient transcripts and reference vectors representing Healthy Controls (HC) and AD patients.
  • Linguistic Baselines: Four traditional metrics—parse tree depth, word error rate (WER), and filler pause frequency/ratio—were included for structural analysis.

The study utilized the ADReSS Challenge 2020 dataset, comprising 156 balanced speech samples (78 HC, 78 AD) from the DementiaBank database.

Results and Discussion

The proposed feature set was evaluated using Random Forest (RF) and XGBoost classifiers. The results demonstrate that the 15 new features significantly outperform traditional linguistic feature sets.

Performance metrics for RF and XGBoost models comparing traditional linguistic features, the new 15-feature set, and the combined set.

As shown in the table, the 15-feature set achieved a peak accuracy of 85.4% with the Random Forest classifier, compared to only 75.0% for the 40 traditional linguistic features. Interestingly, combining all 55 features often led to a decrease in performance (e.g., 80.6% for RF), suggesting that traditional markers may introduce noise that obscures more potent diagnostic signals.

The most predictive markers identified were the "Topic 1 Keyword Hit Rate" (focusing on the children) and "TF-IDF similarity to Healthy Controls." This highlights that content coverage—specifically identifying key visual elements—is a stronger indicator of cognitive health than general grammatical structure.

Conclusion

This study demonstrates that a compact set of 15 explainable features can match or exceed the performance of high-dimensional deep learning models in AD screening. By using LLMs to define "what to look for" rather than "who has the disease," the researchers provide a transparent diagnostic tool that is both computationally efficient and clinically relevant. Future work will focus on testing these features with open-source LLMs to enhance privacy and validating the model on larger, longitudinal datasets.

Summary updated

Abstract

Alzheimer's disease (AD) has become one of the most significant health challenges in an aging society. The use of spoken language-based AD detection methods has gained prevalence due to their scalability due to their scalability. Based on the Cookie Theft picture description task, we devised an explainable and effective feature set that leverages the visual capabilities of a large language model (LLM) and the Term Frequency-Inverse Document Frequency (TF-IDF) model. Our experimental results show that the newly proposed features consistently outperform traditional linguistic features across two different classifiers with high dimension efficiency. Our new features can be well explained and interpreted step by step which enhance the interpretability of automatic AD screening.
BibTeX
@inproceedings{li2024devising,
  author = {Junan Li and Yunxiang Li and Yuren Wang and Xixin Wu and Helen Meng},
  title = {{Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease}},
  booktitle = {ISCSLP},
  year = {2024},
  url = {https://openreview.net/forum?id=DhKOmHQXm5},
}
Figure