FLIQA-AD: a Fusion Model with Large Language Model for Better Diagnose and MMSE Prediction of Alzheimer's Disease

Junhao Chen, Zhiyuan Ding, Yan Liu, Xiangzhu Zeng, Ling WangNAACL 2025 · DOI

Introduction

Early diagnosis of Alzheimer’s Disease (AD) is a critical clinical challenge. Currently, clinicians integrate structural neuroimaging, such as Magnetic Resonance Imaging (MRI), with cognitive assessments like the Mini-Mental State Examination (MMSE) and patient history from Electronic Health Records (EHR). However, automated systems often struggle to integrate these disparate data types effectively. Existing models frequently face "task interference," where optimizing for disease classification degrades the accuracy of cognitive score regression. Furthermore, processing high-resolution 3D MRI volumes within standard Vision Transformer (ViT) architectures is computationally prohibitive.

Methodology: The FLIQA-AD Architecture

To address these challenges, the authors introduce FLIQA-AD (Fusion Language Image Question Answering). This framework reframes AD diagnosis and MMSE prediction as a unified visual question-answering (VQA) task.

The FLIQA-AD architecture integrates a 3D Vision Encoder, ADFormer fusion module, and an LLM decoder to process multimodal clinical data.

The architecture comprises four primary components:

  1. 3D Adapter: This module processes 3D MRI patches, projecting them into an embedding space that preserves spatial relationships without the need for 2D slicing or excessive flattening. It efficiently reduces feature dimensionality from 1,408 to 768, making the model computationally feasible.
  2. Bio-ClinicalBERT: EHR data, including age, gender, and education, are encoded using this domain-specific model, which was pre-trained on extensive medical corpora like PubMed and MIMIC-III.
  3. ADFormer (The Fusion Core): This module utilizes self-attention and cross-attention mechanisms to allow textual EHR "prompts" to guide the interpretation of visual MRI features. This ensures the model focuses on structural brain changes most relevant to the specific patient's demographic profile.
  4. LLM Decoder (FLAN-T5): Instead of simple linear layers, a Large Language Model reasons over the fused multimodal features to generate diagnostic labels or continuous MMSE scores as natural language outputs.

Datasets and Evaluation

The model was rigorously evaluated on the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset, which includes over 8,300 images across three categories: Alzheimer’s (AD), Mild Cognitive Impairment (MCI), and Normal Control (NC). To test the model's robustness and "zero-shot" generalization capabilities, it was also evaluated on the OASIS-2 dataset without any prior fine-tuning on that specific data, providing a rigorous test of its performance across different imaging protocols.

Key Results

FLIQA-AD demonstrates state-of-the-art performance across both classification and regression tasks. On the ADNI dataset, it achieved a 97% overall accuracy for disease identification and a 0.98 AUC. For MMSE score prediction, the model reached a Coefficient of Determination ($R^2$) of 0.90 and a Pearson Correlation of 0.95, indicating an extremely tight fit between predicted and actual cognitive scores.

Performance visualization showing the high overlap between predicted and true MMSE scores across AD, MCI, and NC groups, alongside t-SNE embeddings of ADFormer features.

The model's zero-shot performance on the OASIS-2 dataset (81% accuracy for AD vs. NC) significantly outperformed other multimodal models like MedBLIP, which achieved only 20% in the same setting. As shown in the embedding visualization, the ADFormer output features form distinct clusters for AD, MCI, and NC patients, indicating that the model has learned a highly separable and clinically relevant representation space.

Conclusion and Future Work

FLIQA-AD represents a significant step forward in multimodal medical AI. By treating diagnosis as a reasoning task and leveraging the contextual power of EHR data through the ADFormer, the model overcomes the traditional trade-offs between classification and regression. While the model shows strong generalization, future work will focus on expanding its ability to handle more complex, open-ended clinical queries and further enhancing its robustness against domain shifts in diverse global datasets.

Summary updated

BibTeX
@inproceedings{chen2025fliqa,
  author = {Junhao Chen and Zhiyuan Ding and Yan Liu and Xiangzhu Zeng and Ling Wang},
  title = {{FLIQA-AD: a Fusion Model with Large Language Model for Better Diagnose and MMSE Prediction of Alzheimer's Disease}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-short.49},
}
Figure