Test-retest reliability of modular-relevant analysis in brain functional network

Xuyun Wen, Mengting Yang, Li‐Ming Hsu, Daoqiang ZhangFrontiers in Neuroscience 2022 · DOI

TL;DR

This study systematically evaluates the reliability of analyzing brain functional networks at the mesoscale (modules). The findings show that popular modular-relevant metrics are generally reliable, but their stability is significantly influenced by the choice of module detection algorithm and how brain regions are defined (atlas). This work offers crucial guidance for researchers aiming to extract robust and reproducible insights from brain network data.

What Problem Does This Paper Address?

Understanding the human brain's complex organization often involves modeling it as a network, where brain regions are nodes and their functional connections are edges. Functional Magnetic Resonance Imaging (fMRI) is a key tool for mapping these networks. While researchers have extensively studied global (macroscale) and regional (microscale) network properties, the mesoscale organization – specifically, how brain regions group into functional modules – has received less attention regarding its reliability. Modules are thought to represent specialized subsystems, and metrics derived from them are increasingly used as potential biomarkers for brain diseases. However, without a thorough assessment of their test-retest reliability (consistency across repeated measurements), the findings from modular analyses might not be reproducible, limiting their clinical applicability. This paper aims to fill this gap by rigorously evaluating the reliability of various modular-relevant network metrics and identifying factors that influence this reliability.

Methods

The researchers employed a comprehensive approach to assess the test-retest reliability of modular-relevant analyses using resting-state fMRI (rs-fMRI) data.

  • Brain Network Construction: They constructed functional brain networks using three different brain atlases, varying in granularity (90, 217, and 1024 regions). This allowed them to investigate the impact of node definition on reliability.
  • Module Detection: To capture modular structures, they utilized a combination of individual-level and group-level module detection methods.
  • Individual-level: Four methods were used: Louvain algorithm, Tabu search, spectral optimization, and Infomap. These were applied to each subject's network independently.
  • Group-level: Twelve methods were employed, combining three group-wise construction frameworks (average-based, similarity maximum-based, consensus clustering) with the four module searching approaches.
  • Modular-Relevant Metrics: Five key metrics were calculated based on the detected modular structures:
  • Modularity (overall network segregation)
    • Intra-modular functional connectivity (connectivity within modules)
    • Inter-modular functional connectivity (connectivity between modules)
    • Within-modular degree (a node's importance within its own module)
    • Participation coefficient (a node's importance in connecting to other modules)
  • Reliability Assessment: Test-retest reliability was quantified using:
  • Normalized Mutual Information (NMI) and Rand Index (RI) for assessing the similarity of modular structures between test and retest scans.
  • Intra-class Correlation Coefficient (ICC) for evaluating the reliability of the modular-relevant network metrics.

Datasets

The study utilized the Human Connectome Project (HCP) database's publicly available test-retest dataset. This dataset includes rs-fMRI scans from 45 healthy subjects, with two sessions per subject separated by an average interval of 140 days. Each session comprised two runs.

Key Insights and Ideas

  • Modularity Metrics are Generally Reliable: The study found that popular modular-relevant network metrics (modularity, intra/inter-modular FC, within-modular degree, participation coefficient) demonstrate moderate-to-good test-retest reliability. This suggests that these mesoscale measures can provide robust insights into brain organization.
  • Module Detection Algorithm Matters (Especially Individually): The choice of module detection algorithm significantly impacts reliability. For individual-level analysis, the Louvain algorithm consistently performed best, particularly as network size increased. Infomap, however, showed poorer reliability, potentially due to its sensitivity to edge reliability in fMRI data.
  • Group-Level Analysis Shifts Focus: While individual-level reliability is sensitive to the module searching strategy, group-level reliability is more dependent on the group-wise construction framework. Consensus clustering and average-based frameworks generally yielded more reliable results than similarity maximum-based methods.
  • Atlas Granularity and Generation Influence Reliability: The way brain regions are defined (the atlas) also plays a role. While finer atlases (more regions) can increase the difficulty of module detection and potentially reduce reliability (especially for individual-level analysis), the method of atlas generation (anatomical vs. functional connectivity-based) also seems to influence results, with no single atlas consistently outperforming others across all metrics and analyses.
  • Structural vs. Functional Connectivity: The study hints that regional metrics like within-modular degree and participation coefficient are less reliable in functional networks (fMRI) compared to structural networks (DTI), suggesting that structural connectivity might be inherently more stable.

Key Results

  • Modularity: Consistently showed the highest reliability (good to excellent) across all atlases and methods.
  • Other Metrics: Intra- and inter-modular FC, within-modular degree, and participation coefficient generally exhibited moderate-to-good reliability.
  • Module Detection Algorithms: Louvain algorithm was generally superior for individual-level analysis. For group-level analysis, consensus clustering and average-based frameworks outperformed similarity maximum methods.
  • Brain Atlases: AAL atlas often yielded higher reliability for individual-level modular structure detection, likely due to its smaller number of nodes. However, the impact of atlas on network metric reliability was less pronounced than the module detection algorithm.

Limitations and Future Work

  • Limited Scope of Factors: The study focused on module detection algorithms and node definition approaches. It acknowledges that other factors, such as imaging acquisition parameters, preprocessing steps, and FC estimation methods, also significantly influence reliability and were not explored here.
  • rs-fMRI Specificity: The findings are specific to resting-state fMRI data and the HCP acquisition protocol. Generalizability to other fMRI paradigms or datasets may vary.
  • Multimodal Integration: The authors suggest that a systematic reliability evaluation using multimodal data (e.g., combining fMRI with DTI) from the same population would provide a deeper understanding of the structural and functional architecture of the brain.
  • Voxel-wise Analysis Caution: The paper highlights that voxel-wise analysis in functional brain networks can be less reliable due to the increased complexity and potential for instability, suggesting caution when using such approaches.

Summary updated

Abstract

This paper provides important guidance for choosing reliable modular-relevant network metrics and analysis strategies in future studies.
BibTeX
@inproceedings{wen2022test,
  author = {Xuyun Wen and Mengting Yang and Li‐Ming Hsu and Daoqiang Zhang},
  title = {{Test-retest reliability of modular-relevant analysis in brain functional network}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2022.1000863},
}
Figure