Why Text Outperforms Audio in Automated Cognitive Assessment

60 concepts53 papers1 proposal

When we think about diagnosing neurodegenerative diseases or cognitive impairments from speech, our intuition naturally gravitates toward how a person sounds. We listen for the physical markers of vocal production: tremors, shifts in pitch, acoustic jitter, or long, hesitant pauses.

Yet, a robust consensus across recent machine learning literature reveals a highly counter-intuitive truth: what a person says carries far more diagnostic weight than how they say it. In automated cognitive assessment, linguistic features derived from text transcriptions consistently outperform raw acoustic features.

Understanding why text dominates audio requires looking past the physical mechanics of speech and examining how cognitive decline systematically dismantles the architecture of language.

Quantifying the Performance Gap

The superiority of textual features over acoustic ones is not a marginal statistical anomaly; it is a substantial and repeatable performance gap.

In speech-based Alzheimer's detection, models focusing entirely on linguistic features can outperform acoustic models by over 20% in classification accuracy, as shown by Zehra Shah et al. in Learning Language and Acoustic Models for Identifying Alzheimer's Dementia From Speech. This finding is mirrored by Ning Wang et al. in Modular Multi-Modal Attention Network for Alzheimer's Disease Detection Using Patient Audio and Language Data, where the authors noted that text-based models carry significantly more diagnostic weight than audio-only models. Even when compared to multimodal systems that fuse both audio and text, text-only analysis frequently matches or exceeds the performance of the combined system, as demonstrated by Junwen Duan et al. in CDA: A Contrastive Data Augmentation Method for Alzheimer's Disease Detection.

Why does this gap exist? As Maria Yancheva et al. explain in Using linguistic features longitudinally to predict clinical scores for Alzheimer's disease and related dementias, neurodegenerative conditions like Alzheimer's disease manifest primarily as cognitive-linguistic impairments rather than motor-speech disorders. The disease erodes the cognitive planning required to formulate sentences and retrieve words long before it degrades the physical muscles used to produce vocal sounds.

The Anatomy of Linguistic Biomarkers

To understand why text provides such a clear window into cognitive health, we must look at the specific layers of language that machine learning models extract from transcripts.

The Preservation of Grammar vs. The Loss of Meaning

One of the earliest signs of cognitive decline is a drop in information density. In Vector-space topic models for detecting Alzheimer's disease, Frank Rudzicz et al. discovered that individuals in the early stages of dementia often maintain relatively intact grammatical structures while their ability to convey meaning deteriorates. They use more words to say less. Traditional acoustic analysis misses this entirely, whereas text models capture the thinning density of information.

Psycholinguistic Erosion

Beyond simple word counts, the types of words a speaker selects change predictably. Jieyuan Zheng et al. demonstrated in Exploring MMSE Score Prediction Model Based on Spontaneous Speech that incorporating psycholinguistic attributes—such as the age at which a word is typically acquired or its level of concreteness—significantly enhances a model's ability to detect vocabulary loss. As cognitive function declines, speakers substitute specific, late-acquired nouns with generic, high-frequency words (e.g., saying "thing" instead of "microscope").

High-Level Macro-Descriptors

Modern natural language processing allows models to look beyond individual words to evaluate global discourse structure. Catarina Botelho et al. showed in Macro-descriptors for Alzheimer's disease detection using large language models that Large Language Models (LLMs) are highly effective when used to extract interpretable "macro-descriptors" of speech, such as global coherence and lexical diversity.

Similarly, Jinho D. Choi et al. introduced a meta-semantic representation in Meta-Semantic Representation for Early Detection of Alzheimer’s Disease that analyzes entities and predicate-argument structures. This high-level textual analysis successfully distinguished individuals with mild cognitive impairment (MCI) from healthy controls even when traditional clinical cognitive tests failed to do so.

The Surprising Resilience to Transcription Noise

A common counter-argument to prioritizing text is the dependency on Automated Speech Recognition (ASR). It seems intuitive that errors in an automatically generated transcript would corrupt the downstream diagnostic model.

However, multiple studies show that linguistic features are remarkably resilient to transcription noise. In A Comparative Study of Acoustic and Linguistic Features Classification for Alzheimer's Disease Detection, Jianwei Yu et al. demonstrated that text-based models perform just as well using noisy, automatically generated transcripts as they do with perfect, manual ones. This finding is supported by Jieping Ye et al. in Automatic Detection of Alzheimer's Disease Using Spontaneous Speech Only, which concluded that semantic markers are robust against the typical errors introduced by speech-to-text pipelines.

This resilience occurs because linguistic context is distributed. If an ASR model mishears a specific word, the surrounding syntactic structure, part-of-speech distribution, and vocabulary diversity metrics remain largely intact. Acoustic features, by contrast, can be highly sensitive to environmental noise, microphone quality, and baseline vocal variations.

There is a caveat: in extremely low-resource language scenarios where ASR error rates are catastrophic, the physical properties of the voice can offer a more stable baseline simply because they bypass transcription entirely, as noted by Jaejin Cho in Automatic Detection and Assessment of Alzheimer Disease Using Speech and Language Technologies in Low-Resource Scenarios. But when reasonable transcription is available, text remains the superior signal.

Generalizing Beyond Dementia

The dominance of textual features is not unique to Alzheimer's disease; it extends across a wide variety of clinical and behavioral classification tasks:

By shifting the focus from the physical mechanics of the voice to the structural and semantic integrity of language, text-based models capture the direct downstream consequences of cognitive processing, making text the most powerful tool available for automated spoken language analytics.

Key Foundational Work

Using linguistic features longitudinally to predict clinical scores for Alzheimer's disease and related dementias (2015) — contributed 1 concept elements to this direction.
Applying Discourse Analysis and Data Mining Methods to Spoken OSCE Assessments (2008) — contributed 1 concept elements to this direction.
Stacked auto-encoder for ASR error detection and word error rate prediction (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_textual_feature_dominance_071dca48,
  title = {textual feature dominance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textual-feature-dominance/rdr_071dca480f9b4e6f8dd1f84a2de46493},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_textual_feature_dominance_071dca48_v1,
  title = {textual feature dominance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textual-feature-dominance/rdr_071dca480f9b4e6f8dd1f84a2de46493/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake