Acoustic and Linguistic Biomarkers for Cognitive Impairment Detection from Speech

Catarina Botelho, David Gimeno-Gómez, Francisco Teixeira, John Mendonça, Patrícia Pereira, Diogo A. P. Nunes, Thomas Rolland, Anna Pompili, Rubén Solera-Ureña, Maria Ponte, David Martins de Matos, Carlos-D. Martínez-Hinarejos, Isabel Trancoso, Alberto AbadInterspeech 2025

TL;DR

This paper presents a comprehensive approach to automatically detect cognitive impairment from speech, focusing on the challenging three-class problem of distinguishing healthy individuals, those with Mild Cognitive Impairment (MCI), and dementia patients. By fusing multiple acoustic and linguistic feature sets, including novel LLM-derived macro-descriptors, the authors achieved significant performance improvements, highlighting the power of complementary information for early detection.

What Problem Does This Paper Address?

Cognitive decline, particularly dementia, affects millions worldwide and is a major cause of disability. Early detection is crucial for timely intervention and management. While speech and language impairments are known indicators of cognitive decline, the field faces challenges due to a lack of standardized benchmarks and large, representative datasets for systematic comparison of methodologies. Existing challenges often focus on binary classification (e.g., healthy vs. AD). This paper addresses the more complex, less-studied problem of distinguishing between three distinct cognitive states (healthy, MCI, dementia) using spontaneous speech, as presented in the PROCESS Challenge 2025.

Methods

The authors employed a multi-stage approach:

  • Pre-processing: Speech signals were cleaned using energy-based denoising. Automatic Speech Recognition (ASR) was performed using OpenAI's Whisper (medium and large-v2) and CrisperWhisper, with CrisperWhisper showing better performance for fluency tasks.
  • Feature Extraction: A diverse range of features were extracted:
  • Acoustic: Knowledge-based (KB) features (eGeMAPS, ComParE, Praat, Pauses) and neural embeddings (ECAPA-TDNN, TRILLsson).
  • Linguistic: KB features (Ling, Fluency) and LLM-derived macro-descriptors (Macro) from Llama-3.1-70B-Instruct.
  • Neural Text Embeddings: BERT, RoBERTa, and Longformer.
  • Classification: Various classifiers were used, including Support Vector Machines (SVMs), Decision Trees (DT), Random Forest (RF), Multi-layer Perceptron (MLP), and Fuzzy Fingerprints (FFP).
  • Model Ensembling: A late fusion strategy using logistic regression was employed to combine the predictions of the top-performing individual systems. This aimed to leverage complementary information from different feature sets and models.

Datasets

The primary dataset used was the PROCESS Challenge 2025 dataset. This dataset is designed for three-class classification (Healthy Control, MCI, Dementia) and includes audio recordings from three elicitation tasks: Cookie Theft Description (CTD), Phonemic Fluency (PFT), and Semantic Fluency (SFT). The paper notes that this dataset is less commonly studied than binary classification benchmarks.

Key Insights and Ideas

  • Complementarity is Key: Fusing multiple, diverse feature sets and models significantly outperforms individual systems. This highlights that different aspects of speech capture distinct manifestations of cognitive impairment.
  • Fluency Tasks are Informative: Systems trained on Phonemic Fluency (PFT) and Semantic Fluency (SFT) tasks generally achieved higher performance, suggesting that executive function and lexical access impairments are strongly reflected in speech.
  • LLM Macro-descriptors Shine: Macro-descriptors derived from Large Language Models (LLMs) proved to be highly effective, especially when combined with acoustic features and fluency tasks.
  • Pause Features Matter: Features related to pauses and disfluencies, often overlooked, were found to be highly informative, particularly in the context of PFT and SFT.
  • The Dementia Class is Challenging: Achieving a non-zero F1-score for the dementia class was difficult, likely due to class imbalance and potential age discrepancies in the data.

Key Results

  • The best-performing individual systems achieved a Unweighted Average F1-score (UAF1) of around 54.5% on the development set.
  • Model ensembles, particularly those combining up to six systems, showed substantial performance improvements. The final selected ensembles (Ensemble #1 and Ensemble #2) achieved UAF1 scores of 67.8% and 67.4% respectively on the development set.
  • The F1-score for the dementia class in the best ensemble reached 57.2%, a significant improvement over many individual systems that scored 0.
  • Systems incorporating pause features and LLM macro-descriptors on the PFT task were consistently selected for the best ensembles.
  • Longformer embeddings for the CTD task and combined ECAPA-TDNN/TRILLsson embeddings for all tasks were also highly represented in top ensembles.

Limitations and Future Work

  • Data Scarcity and Imbalance: The PROCESS dataset suffers from significant class imbalance and limited data, especially for the dementia class, which hinders performance and may explain the gap between development and test set results.
  • Demographic Information: The approach did not incorporate demographic data (e.g., age, education), which physicians use in clinical decision-making. This omission, due to data limitations, impacts real-world applicability.
  • Interviewer Influence: The presence of interviewer interventions in speech recordings could influence results. While efforts were made to mitigate this, it remains a challenge.
  • Overfitting Risk: Extensive experimentation on a limited dataset increases the risk of overfitting.
  • Future Work: The authors suggest future research should focus on larger, more balanced datasets and incorporating demographic information. Further investigation into the interpretability of features and classifiers is also warranted.

Summary updated

BibTeX
@inproceedings{botelho2025acoustic,
  author = {Catarina Botelho and David Gimeno-Gómez and Francisco Teixeira and John Mendonça and Patrícia Pereira and Diogo A. P. Nunes and Thomas Rolland and Anna Pompili and Rubén Solera-Ureña and Maria Ponte and David Martins de Matos and Carlos-D. Martínez-Hinarejos and Isabel Trancoso and Alberto Abad},
  title = {{Acoustic and Linguistic Biomarkers for Cognitive Impairment Detection from Speech}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure