On Large Foundation Models and Alzheimer’s Disease Detection

Chuyuan Li, Giuseppe Carenini, Thalia FieldNAACL 2025

TL;DR

This paper investigates the potential of open-source Large Language Models (LLMs) like Llama-3.1 for detecting Alzheimer's Disease (AD) using picture description tasks. The study finds that LLMs can achieve performance comparable to traditional supervised methods with minimal supervision, offering interpretable explanations, while Vision-Language Models (VLMs) like LLaVA-NeXT struggle with this specific task, exhibiting unexpected "neurodegenerative" symptoms themselves.

What Problem Does This Paper Address?

The healthcare sector is increasingly interested in leveraging Large Foundation Models (LFMs), particularly open-source LLMs, for clinical applications. However, there's a significant question about whether these generalist models can match specialist performance without extensive fine-tuning, especially in sensitive areas like neurodegenerative disease detection. Alzheimer's Disease (AD) detection is a critical area where early intervention is key, but some diagnostic modalities are invasive or costly. Language-based assessments, like picture description, offer a non-invasive and easily collectible data source. This paper aims to bridge this gap by evaluating the innate capabilities of LLMs and VLMs for AD detection using this language-based approach, exploring the effectiveness of different prompting strategies.

Methods

The researchers employed two main types of foundation models: a language-only LLM (Llama-3.1-8B) and a Vision-Language Model (VLM) (LLaVA-NeXT-8B). Both models were tasked with analyzing picture descriptions to classify participants as either having Alzheimer's Disease or being healthy controls.

A key aspect of their methodology was prompt engineering. They designed systematic prompts for both models, dividing them into:

  • Background Prompt: Setting the context, role, or providing linguistic cues related to AD.
  • Question Prompt: Directing the model to perform a specific task (e.g., classification) and exploring different reasoning styles like "Short Answer" and "Chain of Thought" (CoT).
  • Example Prompt: For few-shot learning, providing demonstrations of correct input-output pairs.

For LLaVA, an image prompt was also included. They evaluated both zero-shot (no examples) and few-shot (with examples) learning scenarios. The models' outputs were then analyzed for class prediction, probability, and reasoning.

Datasets

The study utilized a dataset from Jang et al. (2021), comprising 63 patients diagnosed with Alzheimer's Disease (or exhibiting initial symptoms of Mild Cognitive Impairments) and 67 healthy controls. Participants completed several tasks, including a picture description task, which was the focus of this research. The speech data was transcribed using WhisperX and manually verified.

Key Insights and Ideas

  • LLMs as nascent AD specialists: Open-source LLMs like Llama-3.1 can perform surprisingly well in detecting AD from picture descriptions, even without task-specific fine-tuning.
  • Prompting is paramount: The choice of background and question prompts significantly impacts LLM performance, shifting the model's decision-making threshold.
  • CoT reasoning enhances moderation: Chain of Thought prompting helps LLMs provide more balanced predictions, evidenced by improved Sensitivity and Specificity.
  • VLMs falter on subtle language: Despite their visual capabilities, VLMs like LLaVA struggled with the linguistic nuances of the picture description task, exhibiting behaviors akin to "neurodegenerative" issues themselves.
  • Zero-shot can be competitive: In some cases, zero-shot prompting with well-crafted prompts can match or even surpass few-shot prompting.

Key Results

  • Llama-3.1 performance: The Llama-3.1 model, with optimal prompting (e.g., Context+Role+Ling background and G. CoT question), achieved an AUC score of 76%, matching or exceeding traditional supervised classifiers like Gaussian Naive Bayes (73%), Logistic Regression (73%), and Random Forest (75%).
  • LLaVA-NeXT performance: LLaVA significantly underperformed Llama, with AUC scores up to 10% lower across various prompting strategies.
  • LLaVA's "hallucinations": When prompted to describe a picture, LLaVA often generated descriptions that included fabricated details or exhibited a lack of understanding of subtle cues, leading supervised classifiers to consistently label its output as "Patient."
  • Few-shot vs. Zero-shot: Few-shot prompting generally improved AUC scores, especially when the background prompt was less informative. However, zero-shot prompting was highly effective in certain scenarios.

Limitations and Future Work

  • LLM Randomness: The inherent randomness in LLM outputs requires multiple runs to ensure robust conclusions. While variations were minimal in most cases, random few-shot prompting showed more significant fluctuations.
  • Model and Dataset Specificity: The study focused on specific LLM and VLM variations and a particular dataset. Further research is needed to understand how different models, architectures, and datasets impact prompt sensitivity.
  • VLM Limitations: VLMs are not yet adept at complex language reasoning tasks like AD detection via picture description. Their tendency to "hallucinate" or exhibit "neurodegenerative" symptoms in their outputs is a significant limitation.

Future work includes expanding methods to conversational interactions and other healthcare-related disease detection tasks. A key challenge for multimodal models is enhancing their compositional capabilities for more intelligent information integration. The authors also suggest exploring alternative methods for demonstration example selection in few-shot learning.

Summary updated

BibTeX
@inproceedings{li2025large,
  author = {Chuyuan Li and Giuseppe Carenini and Thalia Field},
  title = {{On Large Foundation Models and Alzheimer’s Disease Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure