Cross-Linguistic Acoustic Biomarkers of Cognitive Impairment Across Interactive and Single-Speaker Speech Tasks

Kenyu Ikeuchi, Tomoaki Ohtsuki, Michelle Cohn, Alyssa Weakley, Yui Ishihara, Joseph Hoang, Taishiro Kishimoto, Sarah Tomaszewski Farias, Jet M. J. Vonk, Chen-Nee ChuaharXiv 2026 · arXiv

TL;DR

Researchers have developed an interpretable framework to identify acoustic biomarkers of cognitive impairment that hold up across different languages and speech tasks. While temporal features like pauses are universal indicators of decline, other markers—like pitch and speech rate—are highly dependent on the specific language spoken and whether the speaker is describing a picture or engaging in a free-flowing conversation.

What Problem Does This Paper Address?

Detecting dementia and Mild Cognitive Impairment (MCI) early is a global health priority, but clinical assessments are often expensive and hard to scale. Automated speech analysis offers a non-invasive alternative, yet most current research suffers from two major flaws:

  1. Task Tunnel Vision: Most studies rely solely on "Picture Description" tasks, ignoring how cognitive impairment manifests in natural conversation.
  2. The "Black Box" Problem: Many modern approaches use complex transformer models that achieve high accuracy but offer zero clinical interpretability. We know that the model works, but not why.

Furthermore, speech isn't universal. A biomarker that works in English might fail in Mandarin due to linguistic differences. This paper seeks to find which acoustic features are truly "language-agnostic" and which ones are shaped by the task and the tongue.

Methods

The authors proposed a multi-stage, interpretable analysis pipeline designed to strip away noise and focus on clinically relevant features:

  1. Speaker Diarization: Using pyannote.audio to isolate the participant's voice from the interviewer.
  2. Feature Extraction: They extracted a massive set of 255–262 features covering temporal patterns (timing), prosody (pitch/intensity), and spectral qualities (texture of the voice) using toolkits like openSMILE and Praat.
  3. Two-Stage Filtering:
    • Statistical: A Kolmogorov-Smirnov (KS) test to keep only features that showed a distributional difference between healthy and impaired groups.
  4. Expert-Guided: A phonetician reviewed the features to exclude those that were too dependent on specific words (segmental level), ensuring the markers were robust across different vocabularies.
  5. Model-Based Selection: They used Recursive Feature Elimination (RFECV) with a Random Forest classifier to find the smallest, most predictive feature set.
Fig. 1. Overview of the proposed method

Datasets

The study utilized five heterogeneous datasets spanning four languages and four distinct speech tasks:

  • ADReSSo (English): Spontaneous picture description (the "Cookie Theft" image).
  • Ivanova (Spanish): Reading a passage aloud from Don Quixote.
  • Chou (Mandarin): Multiple picture description tasks.
  • PROMPT (Japanese): Free conversation with a psychiatrist.
  • Davis ADRC (English): Autobiographical memory recall (e.g., describing a wedding or a favorite vacation).
Dataset composition and clinical diagnosis

Key Insights and Ideas

  • The Universal Clock: Temporal features (like the duration of medial pauses and the ratio of speaking time to silence) were the most robust biomarkers. Regardless of language or task, cognitive decline consistently disrupts the "timing" of speech.
  • The "Anchoring" Effect on Speech Rate: Speech rate was a strong marker in structured tasks (like reading or picture description) where the content is "externally anchored." However, in spontaneous conversation or memory recall, individual differences in storytelling style made speech rate a less reliable indicator.
  • Pitch is Linguistically Coded:
    • In non-tonal languages (English, Spanish), the variability of pitch (F0) was key.
  • In tonal or pitch-accent languages (Mandarin, Japanese), Chroma-based features (which capture relative pitch structure within an octave) were more dominant. This suggests that cognitive decline affects the linguistic use of pitch differently depending on how a language uses tone to convey meaning.
  • Spectral Features are Fickle: Features like MFCCs (often used in speech recognition) were highly dataset-specific, likely reflecting recording conditions rather than universal disease markers.

Key Results

The researchers demonstrated that they could significantly reduce the number of features—sometimes by more than 60%—without losing classification accuracy.

  • Performance Stability: The "KS + Expert + RFECV" configuration maintained or slightly improved ROC-AUC scores compared to using all features. For example, in the Japanese PROMPT dataset, they achieved an AUC of 0.79 using only 101 selected features.
  • Feature Dominance: As shown in the table below, temporal features remained "High" or "Moderate" in importance across every single dataset, whereas voice quality (jitter/shimmer) was consistently "Low."
Cross-dataset comparison of dominant acoustic features

Limitations and Future Work

While this study provides a roadmap for interpretable multilingual screening, it has notable limitations:

  • Acoustics Only: The study intentionally ignored what was said (lexical/semantic content) to focus on how it was said. Future work should integrate these acoustic markers with linguistic signals.
  • Language Breadth: Only four languages were studied. Expanding to low-resource languages is essential for global equity.
  • Task Consistency: Participants didn't perform multiple tasks within the same dataset, making it harder to perfectly isolate the "task effect" from the "dataset effect."

Ultimately, the findings caution against "blindly" pooling speech data from different languages and tasks, highlighting the need for context-aware AI in dementia care.

Summary updated

Abstract

Multilingual dementia detection is essential for scalable and equitable screening. In recent years, automated speech analysis has emerged as a promising approach for supporting dementia detection by enabling non-invasive and scalable assessment of cognitive function. However, existing studies that use automated speech analysis often rely on one type of task, such as picture description, and use black-box models. These studies often neglect the role of task and language. To address these gaps, we performed a cross-linguistic and cross-task acoustic analysis that integrates a conversational dataset with structured speech tasks across four languages (Spanish, Mandarin, English, Japanese). We applied statistical, linguist-/clinician-guided, and machine learning model-based feature selection to derive compact acoustic feature sets while maintaining classification performance. This enabled systematic comparison of cognitive impairment-relevant features across datasets and tasks. Our results revealed that acoustic features did not generalize uniformly across tasks and languages. Temporal features remained robust across all datasets, consistently capturing timing disruption regardless of language or task. In contrast, speech rate showed strong task dependence -- emerging as informative mainly in single-speaker tasks with externally anchored content and pacing, but becoming less reliable when speech content was highly individualized. Pitch-related features further exhibited language-specific patterns, favoring chroma-related features in tonal languages (e.g., Mandarin) and F0-related features in non-tonal languages. These findings caution against indiscriminate pooling of heterogeneous speech data and underscore the need for interpretable, context-aware approaches to multilingual dementia detection using automated speech analysis.
BibTeX
@misc{ikeuchi2026cross,
  author = {Kenyu Ikeuchi and Tomoaki Ohtsuki and Michelle Cohn and Alyssa Weakley and Yui Ishihara and Joseph Hoang and Taishiro Kishimoto and Sarah Tomaszewski Farias and Jet M. J. Vonk and Chen-Nee Chuah},
  title = {{Cross-Linguistic Acoustic Biomarkers of Cognitive Impairment Across Interactive and Single-Speaker Speech Tasks}},
  year = {2026},
  eprint = {2611.17710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17710},
}
Figure