Are neuroanatomical phenotypes for psychiatric disorders robust? An assessment of the reproducibility of grey matter differences in mental illness

Trang Ngoc Cao, James C. Pang, Mehul Gajwani, Ashlea Segal, Alexander Holmes, Joshua F. Wiley, Sidhant Chopra, Juan Zhou, Chiung-Yu Chen, Fang Ji, Ben J. Harrison, Christopher G. Davey, Toby Constable, Jeggan Tiego, Bree Hartshorn, Jessica Kwee, Mark A. BellgrovemedRxiv 2025 · DOI

TL;DR

This study reveals a concerning lack of reproducibility in neuroanatomical findings for psychiatric disorders, with grey matter differences showing very low consistency across different research sites. While larger sample sizes can improve consistency for schizophrenia, current practices are unlikely to yield robust biomarkers for most psychiatric conditions without significant methodological and diagnostic shifts.

What Problem Does This Paper Address?

For decades, researchers have sought to identify distinct neuroanatomical "fingerprints" for psychiatric disorders using MRI. The hope was that consistent patterns of grey matter volume or thickness differences between patients and healthy controls would emerge, serving as reliable biomarkers for diagnosis, prognosis, and treatment. However, despite thousands of studies, the field has struggled to achieve consensus, with findings often contradicting each other even for the same disorder. This paper tackles the critical question of whether current research practices, relying on independent studies with relatively small sample sizes, can ever lead to robust and reproducible neuroanatomical phenotypes for psychiatric disorders.

Methods

The researchers employed a large-scale, multi-site analysis to assess the consistency of neuroanatomical findings. They collected individual-level data from 59 study sites across five psychiatric disorders (schizophrenia, schizoaffective disorder, autism spectrum disorder, major depressive disorder, and bipolar disorder), totaling 2437 patients and 2065 controls.

Key methodological steps included:

  • Standardized Processing: A uniform image processing pipeline was applied across all sites to ensure comparability of surface-based morphometry (SBM) for cortical thickness and voxel-based morphometry (VBM) for grey matter volume.
  • Cross-Site Consistency (CSC) Calculation: For each disorder, they calculated the product-moment spatial correlation between site-specific difference maps (comparing patients to controls). This metric quantifies how similar the observed brain alterations are across different research sites.
  • Benchmarking: To establish a reference point, they performed the same analysis on data from seven study sites for Alzheimer's disease (AD), a condition known for its well-defined neurodegenerative phenotype.
  • Bootstrapping Analysis: To understand the impact of sample size, they used bootstrapping to simulate studies with varying sample sizes and estimate the sample size required to achieve a correlation of r > 0.5 (considered consistent).
  • Exploration of Influencing Factors: They investigated whether factors like data processing choices (thresholding, smoothing, parcellation), demographic characteristics, and MRI scanner variations influenced CSC.

Datasets

The study utilized data from 59 distinct study sites across 25 different studies for the following psychiatric disorders:

  • Schizophrenia (SCZ)
  • Schizoaffective Disorder (SCA)
  • Autism Spectrum Disorder (ASD)
  • Major Depressive Disorder (MDD)
  • Bipolar Disorder (BD)

Additionally, data from 7 study sites comprising 654 patients and 937 controls with Alzheimer's Disease (AD) were used as a benchmark. The paper does not introduce new datasets or benchmarks but rather aggregates existing ones for a meta-analytical approach.

Key Insights and Ideas

  • The "Reproducibility Crisis" in Psychiatry: The study provides strong quantitative evidence that current practices in neuroimaging research for psychiatric disorders are failing to produce reproducible findings. The low cross-site consistency suggests that the observed brain differences are not robust across different research groups.
  • AD as a Gold Standard (for now): Alzheimer's disease, with its well-established neurodegenerative pathology, serves as a crucial benchmark. The significantly higher consistency in AD findings highlights the potential for robust neuroanatomical phenotypes when the underlying biology is more clearly understood and consistent.
  • Sample Size is Crucial, but Not a Panacea: While increasing sample size is shown to improve consistency, particularly for schizophrenia, it's not a universal solution. For disorders like ASD, MDD, and BD, even very large sample sizes might not be enough to achieve robust findings with current methods, suggesting deeper issues with diagnostic categories or phenotypic heterogeneity.
  • Methodological Nuances Have Minor Impact: Surprisingly, variations in common data processing and analysis choices (like smoothing, thresholding, or parcellation) and differences in demographic/clinical characteristics had a relatively small impact on consistency. This suggests the problem lies deeper than just methodological choices.
  • Diagnostic Constructs as a Potential Bottleneck: The paper hints that the current symptom-based diagnostic categories (like those in the DSM) might be too broad and heterogeneous, obscuring underlying biological signals. This aligns with growing calls for more biologically-informed diagnostic frameworks.

Key Results

  • Low Consistency in Psychiatric Disorders: The median pair-wise cross-site correlation (CSC) for grey matter alterations in psychiatric disorders was very low (r ≤ 0.16).
  • High Consistency in Alzheimer's Disease: In contrast, AD showed a significantly higher median CSC of r = 0.54.
  • Schizophrenia Shows Promise with Scale: Bootstrapping analysis indicated that consistent results (r > 0.5) for schizophrenia could be achieved with study-specific sample sizes exceeding approximately 200 cases and 200 controls.
  • Other Disorders Lag Significantly: Consistent findings for ASD, MDD, and BD may require sample sizes in the order of 1000 or more, and even then, consistency might remain low.
  • Methodological Factors Have Limited Influence: Factors like statistical thresholding, spatial smoothing, and parcellation had only a minor impact on CSC estimates.
  • Demographic/Clinical Factors Showed Little Association: Only two out of 19 tested associations between CSC and site-specific features were statistically significant after correction, indicating these factors are not the primary drivers of inconsistency.

Limitations and Future Work

  • Limited Sample Sizes for Some Disorders: While the study used a large dataset, the maximum sample sizes available for ASD, MDD, and BD were still limited (around 100-250 per group), preventing a definitive assessment of the sample sizes needed for robust findings in these conditions.
  • Potential for Optimistic Bias: The use of a unified processing pipeline might represent an "upper bound" on consistency, as real-world studies often use more varied approaches.
  • Need for Refined Diagnostic Constructs: The authors suggest that future efforts should focus on developing more homogeneous diagnostic subgroups or utilizing transdiagnostic frameworks (like RDoC or HiTOP) to improve the signal-to-noise ratio of neurobiological signals.
  • Importance of Data Sharing and Standardization: The paper advocates for open data sharing and the development of standardized assessment protocols to enable the creation of large, harmonized datasets.
  • Exploring Alternative Modalities: The authors suggest that other imaging modalities (e.g., PET, fMRI) might reveal more consistent phenotypes for certain disorders, as seen with dopamine dysregulation in schizophrenia or brain activity in depression.
  • Unsupervised and Network Approaches: They propose that methods like unsupervised clustering, multivariate analyses, and network-based approaches could be more effective in identifying consistent patterns than traditional group-mean comparisons.

Summary updated

Abstract

Abstract Background Despite thousands of magnetic resonance imaging (MRI) studies reporting grey matter alterations in psychiatric disorders, the field has failed to converge on robust neuroanatomical phenotypes for any specific diagnosis. Here, we examine whether current practices will ever converge on such a phenotype, which is essential for tracking illness risk, progression, and treatment. Methods We evaluated the consistency of brain-wide maps of grey matter volume and cortical thickness alterations obtained for each of 59 study sites of five psychiatric disorders (schizophrenia, schizoaffective disorder, autism spectrum disorder, major depressive disorder, and bipolar disorder), totalling 2437 patients and 2065 controls. We calculated cross-site consistency using spatial correlations between pairs of site-specific difference maps and benchmarked the findings against 7 study sites of Alzheimer’s disease (654 patients, 937 controls). Findings Disorder-specific volume and thickness alterations showed low consistency, with a median pair-wise cross-site correlation of 𝑟 ≤ 0.16 for psychiatric disorders compared to 𝑟 = 0.54 in Alzheimer’s disease. Consistency estimates were not strongly associated with site-specific variations in 19 different demographic, clinical, and scanner characteristics of the study participants and were robust to data processing and analysis. Bootstrapping analyses indicated that consistent results (𝑟 > 0.5) could be obtained for schizophrenia if study-specific sample sizes exceed approximately 200 (for cases and controls), but consistent findings for other disorders may require much larger samples. Interpretation Our findings indicate that current widespread practices, involving case-control comparisons of convenience samples numbering between 30 and 100 patients, will not converge on robust neuroanatomical phenotypes for psychiatric disorders. Increasing sample size will facilitate this goal in schizophrenia, but much larger samples, or refined ascertainment strategies aiming to recruit phenotypically homogeneous patient subgroups, may be required for other disorders.
BibTeX
@misc{cao2025are,
  author = {Trang Ngoc Cao and James C. Pang and Mehul Gajwani and Ashlea Segal and Alexander Holmes and Joshua F. Wiley and Sidhant Chopra and Juan Zhou and Chiung-Yu Chen and Fang Ji and Ben J. Harrison and Christopher G. Davey and Toby Constable and Jeggan Tiego and Bree Hartshorn and Jessica Kwee and Mark A. Bellgrove and Sidhant Chopra},
  title = {{Are neuroanatomical phenotypes for psychiatric disorders robust? An assessment of the reproducibility of grey matter differences in mental illness}},
  year = {2025},
  doi = {10.1101/2025.07.09.25331220},
  howpublished = {medRxiv},
}
Figure