Screening for cognitive decline from a short recorded picture description is cheap to collect but expensive to analyze. The standard pipeline needs a human to transcribe every recording verbatim. A large language model (LLM) that reads automatic speech recognition (ASR) output could remove that step, especially through in-context learning (ICL): the model is shown a few labeled examples inside its prompt and classifies a new case without any weight updates. The usual way to choose those examples is to retrieve the training cases whose text is most similar to the new one. This proposal argues that the usual recipe fails in a predictable way in clinical speech. Every participant in the main benchmarks describes the same picture, the "Cookie Theft" scene. Text similarity therefore mostly measures which parts of the picture someone mentioned, and which ASR errors they share. It does not measure how impaired their language is.
The fix is to change what "similar" means. We ask an LLM to expand the bare labels "Alzheimer's disease" and "healthy control" into concrete, checkable markers of impaired speech. Examples are many long pauses, vague pronouns in place of nouns, repeated phrases, and few distinct picture details mentioned. Each ASR transcript is scored on these markers, and we retrieve demonstrations that are close in this marker space rather than in a general sentence-embedding space. The expanded descriptions do two jobs: they define the retrieval key, and they appear in the prompt as label definitions. The question is whether this lets a frozen LLM match supervised models trained on ASR text. If it does, ICL becomes a credible route around the manual transcription bottleneck. If a simple classifier on the same markers does just as well, we will report that plainly.
Core Claim
On same-prompt clinical speech, the nearest neighbors found by a sentence encoder are close to chance in diagnostic agreement. Retrieving demonstrations by distance in an LLM-generated clinical-attribute space raises neighbor label agreement and ICL accuracy on ASR-only input. The prediction is not that ICL beats every classifier on that attribute vector. The prediction is narrower. Attribute-space retrieval should close most of the gap between plain similarity-based ICL and the strongest fine-tuned ASR-text systems. It should also degrade less than similarity-based retrieval as the input moves from manual to ASR transcripts.
Formally, let $D_1,\dots,D_m$ be the $m$ marker descriptions produced by label expansion. Let $a(x)\in\mathbb{R}^m$ be the vector of LLM-assigned scores of transcript $x$ against them, standardized on the training fold. Demonstrations for a query $q$ are the $k$ training transcripts $x_i$ minimizing
$$d(q,x_i)=\lVert W\,(a(q)-a(x_i))\rVert_2,$$
where $W$ is a diagonal weighting. In the main variant $W$ is the identity. In a second variant each weight is the attribute's discriminativeness measured on the training fold only.
Grounding in Prior Work
Three results motivate the design. Active Learning Principles for In-Context Learning with Large Language Models compared ways of choosing demonstrations across 24 text tasks. It found that nearest-neighbor retrieval with a sentence encoder (the method known as KATE) beats random, diversity and uncertainty selection, enough that a 124M-parameter GPT-2 with retrieved examples outperformed a 20B GPT-NeoX with random ones. The same study found that correct labels on retrieved demonstrations matter a great deal. That cuts both ways for us. It says retrieval is the lever worth pulling, but only if neighbors are informative about the label, and that is exactly what we doubt holds for Cookie Theft transcripts. Making Pre-trained Language Models Better Few-shot Learners used the same kind of sentence-similarity demonstration selection inside prompt-based fine-tuning and reported an 11% average gain over standard fine-tuning. Both studies used clean human-written text with topically diverse inputs. Our setting removes both assumptions.
On the speech side, Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks is the closest existing work. It feeds flattened ASR hypothesis graphs to LLMs with zero- or one-shot prompts and closes 81% of the gap to an n-best oracle on spoken question answering. It uses fixed demonstrations, bare categorical labels and short utterances, and it breaks down above 70% word error rate. We keep its premise that LLMs can read imperfect ASR output, and change the demonstration choice and the label representation. Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks is a constraint. A roughly 150M-parameter speech-token model showed no ICL ability until meta-trained, and it works only on 50-token clips. This is why we route through ASR text and a text LLM rather than raw audio.
The method for keeping acoustic information comes from Alzheimer Disease Classification through ASR-based Transcriptions: Exploring the Impact of Punctuation and Pauses. That work writes pauses found by forced alignment into Whisper transcripts as punctuation, and this lifted a biLSTM classifier from 0.750 to 0.833 accuracy on ADReSS, against 0.854 on manual transcripts. We reuse that encoding, so pause-related markers are visible to the LLM. Using the Outputs of Different Automatic Speech Recognition Paradigms for Acoustic- and BERT-Based Alzheimer's Dementia Detection Through Spontaneous Speech supplies the strongest speech-only number and a warning. Multiple-hypothesis BERT models reached about 97% on the evaluation set but 84.51% on the ADReSSo test set. Small clinical cohorts reward overfitting, so all our design choices are made inside training folds.
Label expansion comes from vision. DesCo: Learning Object Recognition with Rich Language Descriptions and Prompt, Generate, Then Cache both ask GPT-3 to unpack class names into attribute descriptions, which improves detection and few-shot classification respectively. DesCo also documents the main risk. Models match the nouns in long descriptions and ignore the qualifiers, and descriptions from weaker generators hurt. We use the expansion differently: as a set of separately scored axes for retrieval, not only as a longer label string. That lets us test whether each axis actually separates the classes, instead of trusting the generator.
Research Program
The pipeline has four stages. First, recordings from ADReSS and ADReSSo (Alzheimer's Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge; Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge) are transcribed with Whisper-large. Pauses are inserted as punctuation following Gómez-Zaragozá et al.
Second, one fixed instruction-tuned open-weight LLM is asked to expand the two labels into a candidate pool of speech markers, each written as one checkable sentence. The same LLM then rates every transcript on every marker. Any marker whose train-fold discriminativeness is near zero is dropped. This filter is the guard against hallucinated or bag-of-words markers.
Third, each test transcript retrieves $k$ balanced neighbors by the distance above. The prompt contains the expanded label definitions, the retrieved transcripts with their gold labels, and the query.
Fourth, we analyze where it works. We compute the diagnostic agreement of retrieved neighbors under each similarity space. We measure how marker scores shift between manual and ASR transcripts of the same ADReSS speaker, to find which markers survive recognition errors. We also break down errors by speaker word error rate. ADReSS has both manual and ASR transcripts, which allows controlled noise comparisons. ADReSSo is speech-only and serves as the realistic transcription-free test.
Baselines
The naive reference points are a majority-class prediction and ICL with randomly chosen, class-balanced demonstrations under bare labels. The second shows what a frozen LLM does with no retrieval and no expansion.
The obvious ICL competitor is KATE-style retrieval from Active Learning Principles for In-Context Learning with Large Language Models. It retrieves neighbors by Sentence-BERT cosine similarity over the same pause-augmented ASR transcripts, with the same LLM, $k$, prompt format and label definitions. Because it shares every component except the similarity space, the comparison isolates the key ingredient. To beat it, our method must improve accuracy on the official test sets and in repeated stratified cross-validation, with non-overlapping bootstrap intervals over test speakers.
The strongest existing systems are supervised. On ADReSS, the biLSTM with pause punctuation of Alzheimer Disease Classification through ASR-based Transcriptions reports 0.833 accuracy on ASR transcripts and 0.854 on manual ones. To BERT or Not To BERT reports 83.3% for fine-tuned BERT. On ADReSSo, Using the Outputs of Different ASR Paradigms reports 84.51% test accuracy for its linguistic-only model, against a challenge baseline of 78.87%. We will re-run a fine-tuned BERT on our exact pause-augmented transcripts and splits, with a tuning budget matched to the number of prompt and marker configurations we try, so differences are not due to transcription. Matching 0.833 on ADReSS and 84.51% on ADReSSo without gradient updates counts as success. Exceeding them counts as beating them.
Three ablations isolate the ingredients. We swap the marker space for SBERT space while keeping the expanded definitions. We keep the marker space but use bare labels. We use the LLM's own hidden states as the retrieval space, following the internal-embedding idea in Adapters Selector.
Why not just…?
Why not just fine-tune BERT on the ASR transcripts? This is a serious competitor. The cited results put it at 83.3% on ADReSS, and the 97%-to-84.51% evaluation-to-test drop in Pan et al. shows how brittle it is on cohorts of around a hundred speakers. We run it head-to-head under identical splits. The argument for ICL rests on lower variance across splits and on needing no retraining when new clinics or tasks arrive. Accuracy parity alone would not justify it, so we report split-to-split variance as a primary outcome alongside mean accuracy.
Why not just train logistic regression on the attribute scores, or take a majority vote of the attribute-space neighbors? This is the biggest threat. If the marker vector carries all the signal, the LLM's reading of the demonstrations is unnecessary. We run both on the same vectors. If either matches the full ICL system, the contribution reduces to "LLM-generated, ASR-robust clinical features," and we will say so. ICL earns its place only if it beats both, which would mean the model uses demonstration text beyond the marker scores.
Why not just prompt zero-shot with the expanded label definitions and skip retrieval? DesCo's bag-of-words finding suggests long definitions alone invite keyword matching. Margatina et al. suggest well-chosen demonstrations matter more than model scale. Still, a strong LLM with good definitions might suffice. The zero-shot definition-only condition is part of the ablation grid, and the comparison will settle it directly.
Proposed Experiments
The first decisive check is cheap and comes before any LLM classification. Within the ADReSS training set, under leave-one-speaker-out, we measure how often a transcript's top-$k$ neighbors share its diagnosis in SBERT space versus marker space. If SBERT agreement is already well above chance, the central premise is wrong, and the project reduces to testing label expansion alone.
The second check runs the full grid on ADReSS: random, SBERT and marker retrieval, each with bare or expanded labels, plus zero-shot, logistic regression, neighbor vote and fine-tuned BERT. Each is evaluated on manual and on ASR transcripts. Success means marker retrieval shows a smaller manual-to-ASR drop than SBERT retrieval and lands within the supervised range above.
The third check is a single locked run on the ADReSSo test set, with markers, $k$ and weighting fixed beforehand on training folds. It is compared against 84.51%. A secondary run predicts MMSE (Mini-Mental State Examination) scores by having the LLM read retrieved neighbors' scores, reported against the ADReSSo baseline RMSE of 5.28 and the ADReSS linguistic baseline of 5.20.
The fourth check is a stratified analysis by speaker word error rate. It asks whether marker retrieval also extends the noise range in which LLM reading of ASR output stays useful, given the 70% breakdown point reported in the word-confusion-network study.
References
- Active Learning Principles for In-Context Learning with Large Language Models
- Making Pre-trained Language Models Better Few-shot Learners
- Towards ASR Robust Spoken Language Understanding Through in-Context Learning with Word Confusion Networks
- Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
- Alzheimer Disease Classification through ASR-based Transcriptions: Exploring the Impact of Punctuation and Pauses
- Using the Outputs of Different Automatic Speech Recognition Paradigms for Acoustic- and BERT-Based Alzheimer’s Dementia Detection Through Spontaneous Speech
- DesCo: Learning Object Recognition with Rich Language Descriptions
- Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
- Alzheimer’s Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge
- Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge
- To BERT or not to BERT: Comparing Speech and Language-Based Approaches for Alzheimer's Disease Detection
- Adapters Selector: Cross-domains and Multi-tasks LoRA Modules Integration Usage Method