Detecting Neurocognitive Disorders through Analyses of Topic Evolution and Cross-modal Consistency in Visual-Stimulated Narratives

Jinchao Li, Yuejiao Wang, Junan Li, Jiawen Kang, Bo Zheng, Ka Ho Wong, Brian Mak, Helene H. Fung, Jean Woo, Man-Wai Mak, Timothy Kwok, Vincent Mok, Xianmin Gong, Xixin Wu, Xunying Liu, Patrick C. M. Wong, Helen Meng

TL;DR

This paper introduces a novel framework for early detection of neurocognitive disorders (NCDs) by analyzing the high-level "macrostructure" of speech during visual storytelling tasks. By tracking how topics evolve over time (using a Dynamic Topic Model) and measuring how well speech aligns with visual prompts (via a Text-Image Temporal Alignment Network, or TITAN), the researchers demonstrate that global narrative coherence is a far more powerful biomarker for cognitive decline than traditional, localized language features.

What Problem Does This Paper Address?

When clinicians evaluate patients for neurocognitive disorders (NCDs) like Alzheimer's disease, traditional methods—such as neuroimaging or spinal fluid analysis—are often invasive, expensive, and difficult to scale. Automated speech analysis offers a non-invasive alternative, particularly through Visual-Stimulated Narrative (VSN) tasks, where a participant describes a picture or tells a story from a picture book.

Historically, automated VSN analysis has focused on microstructural features: local, bottom-up metrics like vocabulary diversity, syntax complexity, or the frequency and duration of pauses. While useful, these features fail to capture macrostructural impairments—the top-down, concept-driven cognitive abilities required to maintain a logical storyline, sequence events correctly, and stay contextually relevant to the visual prompt. This paper addresses the challenge of mathematically quantifying these abstract, macrostructural properties of speech to improve early NCD detection.

Methods

The authors propose two distinct macrostructural approaches to analyze VSNs:

  1. DTM-based Temporal Analysis: Using a Dynamic Topic Model (DTM), the framework splits a participant's transcript into sequential time slices to track how topics shift across the narrative arc. From this, they derive six statistical metrics, including topic consistency (how well the words align with expected themes over time), topic variability, and topic change rate.
  2. Text-Image Temporal Alignment Network (TITAN): To evaluate cross-modal consistency, the authors designed TITAN, a deep learning architecture that processes both the visual stimuli (images) and the spoken narrative (text) simultaneously. TITAN uses pretrained vision-language models (like CLIP) to project images and text into a shared semantic space. It then leverages a Self-Attention mechanism enhanced with Rotary Position Embedding (RoPE) to capture temporal dependencies and flag misalignments—such as when an NCD patient misidentifies an object or skips a scene entirely.
Fig. 2. (a) The proposed TITAN model. (b) The Attention module of TITAN. (c) The correlation map between image and text embeddings in TITAN, where the 15 textual chunks are manually segmented and aligned with the corresponding images. TITAN takes images and text narratives as inputs to predict NCDs, with the Attention mechanism using RoPE to emphasize important positions.

Datasets

The framework was evaluated across three distinct datasets:

  • CU-MARVEL-RABBIT: A newly introduced Cantonese corpus collected by the authors' research group in Hong Kong. It features 758 participants over the age of 60 (467 healthy controls, 291 NCD patients) performing a sequential storytelling task based on a 15-frame cartoon book ("Rabbit Story").
  • ADReSS & ADReSSo: Two widely used, public English benchmarks based on the static "Cookie Theft" picture description task, containing speech recordings from elderly adults.

Key Insights and Ideas

  • The "Returning Home" Phenomenon: One of the most fascinating findings from the DTM analysis was that healthy controls frequently exhibited a cyclical narrative pattern. Even though the final frames of the "Rabbit Story" did not depict the characters' house, healthy participants often concluded by stating that the characters "returned home," demonstrating a top-down cognitive ability to bring a story to a satisfying closure. NCD patients rarely did this.
  • Chaotic Topic Drift: Healthy controls showed smooth transitions with distinct, consecutive peaks in topic proportions that matched the chronological order of the images. In contrast, the topic distributions of NCD patients were erratic and overlapping, frequently mixing up elements from different scenes.
Fig. 1. The topic distribution for (a) the HC group and (b) the NCD group; and the topic evolution for (c) the HC group and (d) the NCD group; as obtained from DTM. Plots (a) and (b) show the weights of the top-10 topic words for each topic (A to E) for the group. Plots (c) and (d) display the average topic proportions with curves and standard deviations with shaded areas. Ordering of topics A to E is based on the positions of their first peaks in topic evolutions.
  • Macro Beats Micro: Feature contribution analysis using SHAP (Shapley Additive exPlanations) values revealed that macrostructural features—specifically topic variability, topic change rate, and topic consistency—were the most significant contributors to the classification model's decisions, vastly outperforming traditional acoustic and linguistic microfeatures.

Key Results

  • Superior Classification Performance: The TITAN model achieved state-of-the-art or highly competitive results across all three corpora, scoring an F1-score of 0.8889 on ADReSS, 0.8504 on ADReSSo, and 0.7238 on the complex CU-MARVEL-RABBIT dataset.
  • Macro vs. Micro Ablation: On the Cantonese corpus, a Support Vector Machine (SVM) trained purely on macrostructural features significantly outperformed one trained on microstructural features (F1 of 0.6947 vs. 0.5843).
  • Ablation Insights: Removing the image embeddings from TITAN caused the most drastic drop in performance (F1 dropped from 0.7238 to 0.6250), proving that cross-modal text-image alignment is critical for accurate NCD screening.

Limitations and Future Work

While the study demonstrates the power of macrostructural analysis, the authors note a few areas for improvement. The front-end pipeline relies on Automatic Speech Recognition (ASR), word segmentation, and part-of-speech tagging, which can introduce errors—especially when dealing with colloquial or fragmented Cantonese speech.

Future work will focus on optimizing these technical components, extending the framework to incorporate additional modalities (such as video or acoustic prosody directly into the deep learning model), adapting the models for multilingual contexts, and testing their efficacy on other speech- and language-related cognitive disorders.

Summary updated

Abstract

Early detection of neurocognitive disorders (NCDs) is crucial for timely intervention and disease management. Given that language impairments manifest early in NCD progression, visual-stimulated narrative (VSN)-based analysis offers a promising avenue for NCD detection. Current VSN-based NCD detection methods primarily focus on linguistic microstructures (e.g., lexical diversity) that are closely tied to bottom-up, stimulus-driven cognitive processes. While these features illuminate basic language abilities, the higher-order linguistic macrostructures (e.g., topic development) that may reflect top-down, concept-driven cognitive abilities remain underexplored. These macrostructural patterns are crucial for NCD detection, yet challenging to quantify due to their abstract and complex nature. To bridge this gap, we propose two novel macrostructural approaches: (1) a Dynamic Topic Model (DTM) to track topic evolution over time, and (2) a Text-Image Temporal Alignment Network (TITAN) to measure cross-modal consistency between narrative and visual stimuli. Experimental results show the effectiveness of the proposed approaches in NCD detection, with TITAN achieving superior performance across three corpora: ADReSS (F1=0.8889), ADReSSo (F1=0.8504), and CU-MARVEL-RABBIT (F1=0.7238). Feature contribution analysis reveals that macrostructural features (e.g., topic variability, topic change rate, and topic consistency) constitute the most significant contributors to the model's decision pathways, outperforming the investigated microstructural features. These findings underscore the value of macrostructural analysis for understanding linguistic-cognitive interactions associated with NCDs.
BibTeX
@article{li2025detecting,
  author = {Jinchao Li and Yuejiao Wang and Junan Li and Jiawen Kang and Bo Zheng and Ka Ho Wong and Brian Mak and Helene H. Fung and Jean Woo and Man-Wai Mak and Timothy Kwok and Vincent Mok and Xianmin Gong and Xixin Wu and Xunying Liu and Patrick C. M. Wong and Helen Meng},
  title = {{Detecting Neurocognitive Disorders through Analyses of Topic Evolution and Cross-modal Consistency in Visual-Stimulated Narratives}},
  journal = {IEEE Journal of Selected Topics in Signal Processing},
  year = {2025},
  doi = {10.1109/jstsp.2025.3622049},
}
Figure