Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity

Jiawen Kang, Kun Li, Dongrui Han, Jinchao Li, Junan Li, Lingwei Meng, Xixin Wu, Helen Meng

TL;DR

Current machine learning approaches for automated Alzheimer’s Disease (AD) screening rely on "black-box" pattern recognition, which often exploits statistical shortcuts rather than measuring actual cognitive decline. This paper introduces Agentic Cognitive Profiling (ACP), a multi-agent LLM framework that mirrors clinical protocols by decomposing cognitive tests into atomic tasks, using deterministic function calling for error-free scoring, and generating fully auditable diagnostic reports. ACP achieves 85.3% accuracy in AD screening, proving that clinical validity and high predictive performance can coexist without compromise.

What Problem Does This Paper Address?

Most automated AI systems for detecting Alzheimer's Disease treat the problem as a standard text or audio classification task. They feed speech transcripts or audio features into Pre-trained Language Models (PLMs) like BERT to map inputs directly to a diagnostic label.

The authors argue that this approach suffers from a lack of clinical construct validity—meaning the models aren't actually measuring the underlying neurocognitive deficits (like memory loss or executive dysfunction). Instead, they often fall prey to the "Clever Hans" effect, picking up on confounding proxy features like regional dialects, acoustic shortcuts, or speech recognition errors to make predictions. This makes them brittle, prone to hallucinations, and entirely uninterpretable to clinicians who require verifiable behavioral evidence to make diagnostic decisions.

Methods

To bridge the gap between clinical logic and AI, the authors propose Agentic Cognitive Profiling (ACP), a framework that operationalizes standard clinical workflows using a multi-agent architecture.

Figure 2: Overview of the Agentic Cognitive Assessment Framework. The workflow comprises three stages: (1) Administration: Collection of standardized task responses; (2) Examination: Multi-agent Workflow with Deterministic Function Calling and Verification Loop; (3) Inference: Aggregation of verified metrics for classification and explainable reporting.

The workflow is divided into three distinct stages:

  1. Administration: Standardized verbal responses and demographic data (age, education) are collected.
  2. Examination (The Multi-Agent Core):
  3. An Assigner routes different parts of the test transcript to specialized, task-specific Examiner Agents (powered by Qwen3-8B).
  4. Crucially, the framework decouples semantic understanding from measurement. While the LLM handles the messy nuances of natural language (disfluencies, colloquialisms), all actual scoring is delegated to Deterministic Function Calls (e.g., list_length(), keyword_check()). This eliminates numerical hallucinations.
  5. A Verifier Agent cross-checks the Examiner's output against the raw transcript in a loop (up to 3 retries) to catch and correct any logic errors.
  6. Inference: A Meta Analyst Agent aggregates the verified scores, normalizes them against demographic tables, and passes them to a classifier (or applies clinical thresholds for a zero-shot approach). It outputs both a binary prediction and a structured, domain-by-domain cognitive profile report.

Datasets

Because no public datasets contain structured responses across multiple cognitive domains, the authors evaluated their framework on an in-house Cantonese speech corpus:

  • Participants: 402 clinically annotated profiles selected from a larger cohort of 1,063 older adults.
  • Tasks: Responses spanned 8 structured cognitive tasks across two gold-standard batteries: the MoCA-SL (Spoken Language subset of the Montreal Cognitive Assessment) and the HKLLT (Hong Kong List Learning Test).
  • Splits: 334 participants for training (used only for the supervised baseline comparison) and 68 for testing, with balanced age and education distributions between healthy controls and AD-risk groups.

Key Insights and Ideas

  • Math and Language Don't Mix in LLMs: LLMs excel at parsing colloquial Cantonese and filtering out speech disfluencies, but they struggle with reliable counting. Delegating quantification to deterministic code functions while letting the LLM handle semantics is the key to achieving clinical-grade reliability.
  • Clinical Logic is a Powerful Prior: Standard machine learning assumes you need massive datasets to learn patterns from scratch. This paper shows that hardcoding clinical protocol logic into an agentic workflow allows a model to perform exceptionally well zero-shot, completely outperforming deep learning models trained on the data.
  • Interpretability via Primitives: By breaking down a diagnostic score into "scoring primitives" (e.g., specific words recalled, exact subtraction errors), the framework creates an explicit audit trail.

Key Results

The ACP framework was evaluated on both its ability to score tests accurately and its ability to predict Alzheimer's.

  • Scoring Accuracy: The framework achieved a 90.5% weighted-average Score Match Rate (SMR) compared to human clinical annotators, with low-inference tasks like Picture Naming and Digit Span scoring above 97%.
  • AD Prediction Accuracy:
  • In a supervised setting (using an SVM trained on the framework's extracted primitives), ACP achieved 85.3% accuracy, outperforming standard BERT (79.4%) and RoBERTa (75.0%) baselines.
  • In a zero-shot setting (applying standard clinical cutoffs directly to the automated scores), ACP achieved 82.4% accuracy, outperforming all baseline models without using a single byte of training data.
Feature importance analysis using SHAP values

A SHAP feature importance analysis (Figure 6) confirmed that the model's decisions were driven by clinically valid markers—primarily episodic memory recall features (n_hkllt4_recall)—rather than arbitrary statistical shortcuts.

Limitations and Future Work

  1. Dependence on Predefined Rules: The framework is strictly bound to existing clinical scoring guidelines. It cannot discover novel, subtle linguistic markers of cognitive decline that aren't already captured by standard tests.
  2. LLM Capability Ceiling: The system's accuracy is ultimately capped by the base LLM’s ability to understand complex or highly ambiguous semantic responses (e.g., abstract reasoning tasks).
  3. Dataset Restrictions: Due to strict privacy and IRB constraints on medical data, the Cantonese evaluation corpus cannot be publicly released, which limits direct reproducibility by outside teams.

Summary updated

Abstract

Automated Alzheimer's Disease (AD) screening has predominantly followed the inductive paradigm of pattern recognition, which directly maps the input signal to the outcome label. This paradigm sacrifices construct validity of clinical protocol for statistical shortcuts. This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic across multiple cognitive domains. Rather than learning opaque mappings from transcripts to labels, the framework decomposes standardized assessments into atomic cognitive tasks and orchestrates specialized LLM agents to extract verifiable scoring primitives. Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling, thereby mitigating hallucination and restoring construct validity. Unlike popular datasets that typically comprise around a hundred participants under a single task, we evaluate on a clinically-annotated corpus of 402 participants across eight structured cognitive tasks spanning multiple cognitive domains. The framework achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines while generating interpretable cognitive profiles grounded in behavioral evidence. This work demonstrates that construct validity and predictive performance need not be traded off, charting a path toward AD screening systems that explain rather than merely predict.
BibTeX
@inproceedings{kang2026agentic,
  author = {Jiawen Kang and Kun Li and Dongrui Han and Jinchao Li and Junan Li and Lingwei Meng and Xixin Wu and Helen Meng},
  title = {{Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity}},
  booktitle = {LLA @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=eq9AEZh9zr},
}
Figure