Challenges in multi-task learning for fMRI-based diagnosis: Benefits for psychiatric conditions and CNVs would likely require thousands of patients

Annabelle Harvey, Clara Moreau, Kuldeep Kumar, Guillaume Huguet, Sebastian Urchs, Hanad Sharmarke, Khadijé Jizi, Charles-Olivier Martin, Nadine Younis, Petra Tamer, Martineau Jean‐Louis, Pierre Orban, Ana Isabel Silva, Jeremy Hall, Marianne van den Bree, Michael J. Owen, David E.j. Linden, Sarah Lippé, Carrie E. Bearden, Guillaume Dumas, Sébastien Jacquemont, Pierre Bellec

TL;DR

This paper investigates the potential of Multi-Task Learning (MTL) to improve diagnostic accuracy for psychiatric conditions and genetic variants using fMRI data. While MTL showed promise for predicting age and sex across different data collection sites, it surprisingly harmed diagnostic accuracy when applied to a combined set of psychiatric conditions and CNVs, suggesting that large sample sizes are crucial for successful MTL in this domain.

What Problem Does This Paper Address?

Machine learning (ML) models show promise for automatically diagnosing psychiatric conditions from fMRI data. However, a major hurdle is the poor generalization of these models to independent datasets, often leading to a sharp drop in performance. This is partly due to the significant biological heterogeneity within current diagnostic categories, which are based on symptoms alone. The authors propose that MTL, by learning multiple related tasks simultaneously, could leverage shared neuroimaging signatures and genetic information (like Copy Number Variants or CNVs) to improve diagnostic accuracy. They aim to explore if MTL can overcome the limitations of small sample sizes and improve diagnostic performance for psychiatric conditions and CNVs.

Methods

The study employs a Multi-Layer Perceptron (MLP) architecture for its MTL models, specifically a "shared bottom" approach where initial layers are shared across tasks, and then the network branches into task-specific heads. They also explore variations like MLPconn_deeper, MLPconcat, CNN, and SM models.

For predicting age and sex across different data collection sites, each site was treated as a separate task. For diagnosing psychiatric conditions and CNVs, each condition was treated as a task.

To establish a baseline, standard single-task learning models (MLP, SVC, LR, Ridge) were used to predict each condition independently.

Datasets

The study utilizes a large resting-state fMRI (rs-fMRI) dataset compiled from multiple studies, including:

  • Clinical genetic datasets: Four cohorts focusing on individuals with CNVs associated with psychiatric disorders (Simons Variation in Individuals Project (SVIP), DEFINE Neuropsychiatric-CNVs Project, UCLA 22q11.2 CNV project, and the Montreal rare genomic disorder family project).
  • Psychiatric conditions cohorts: Five datasets including individuals with Autism Spectrum Disorder (ASD), Attention-Deficit/Hyperactivity Disorder (ADHD), Schizophrenia (SZ), and Bipolar Disorder (BIP) (ABIDE1, ABIDE2, ADHD-200, CNP, and SZ studies).
  • Unselected population: A large dataset from the UK Biobank, used for age and sex prediction experiments.

This dataset comprises a total of 2872 participants across 53 sites, covering 7 CNVs and 4 psychiatric disorders.

Key Insights and Ideas

  • MTL for site harmonization is effective: When tasks are similar (predicting age/sex) and differ primarily due to data collection sites, MTL can indeed improve prediction accuracy, especially with larger sample sizes. This suggests MTL can help overcome site-specific biases.
  • Sample size is paramount for MTL: The effectiveness of MTL is highly dependent on sample size. For age prediction, MTL was detrimental for small sample sizes (<500) but beneficial when sample sizes were large (>1500). This highlights that MTL's ability to pool data is only advantageous when there's enough data to begin with.
  • MTL struggles with heterogeneous diagnostic tasks: Surprisingly, applying MTL to a combined set of psychiatric conditions and CNVs harmed diagnostic accuracy. This suggests that the inherent heterogeneity and potentially weak or non-existent shared biological factors across these diverse diagnostic categories prevent MTL from learning effectively.
  • Task relationships are not stable: Pairwise MTL analysis revealed that the relationships between diagnostic tasks were not consistent across different model architectures. This implies that the "shared latent factors" hypothesis might be too simplistic for complex psychiatric conditions and CNVs, and that sample size and individual task performance are stronger drivers of MTL success.
  • Thousands of patients needed for diagnostic MTL: The study's findings strongly suggest that developing a successful MTL model for diagnosing psychiatric conditions using fMRI would require datasets with thousands of patients across different diagnoses, far beyond what was available in this study.

Key Results

  • Age/Sex Prediction (UK Biobank):
    • For sex prediction, MTL across three UKBB sites showed gains over single-task learning, especially with larger sample sizes (N > 7500).
  • For age prediction, MTL was beneficial only when increasing sample size through site pooling (N > 1500 total subjects). For smaller sample sizes, single-task and multi-task models performed similarly.
  • When applied across a varied collection of sites, MTL improved sex prediction accuracy in most sites and significantly outperformed single-task learning for age prediction (mean loss reduced from 12.2 to 8.9 years²).
  • Diagnostic Accuracy (CNVs & Psychiatric Conditions):
    • Single-task ML models achieved reasonable diagnostic accuracy for some conditions (e.g., DEL 22q11.2 up to 90%, SZ, BIP, DEL 16p11.2, etc., over 70%).
  • MTL applied to the combined set of 7 CNVs and 4 psychiatric conditions decreased overall prediction accuracy compared to single-task learning.
  • Pairwise MTL analysis showed that conditions with high single-task accuracy but small sample sizes (e.g., DEL 22q11.2, DEL 16p11.2) suffered when trained with partners. Larger sample size conditions (SZ, ADHD, ASD) were less affected.

Limitations and Future Work

  • Sample Size Limitation: The most significant limitation is the relatively small sample size for many of the CNV and psychiatric condition cohorts, which the authors argue is the primary reason MTL failed to improve diagnostic accuracy. They suggest that thousands of patients per condition are likely needed.
  • Heterogeneity of Tasks: The authors acknowledge that the tasks (diagnoses) might be too heterogeneous to benefit from joint learning. Future work could explore approaches that focus on finding homogeneous subtypes within conditions.
  • Data Harmonization: While MTL showed benefits for site harmonization, the paper notes that further research into data harmonization techniques is ongoing.
  • Model Architectures: While they explored several MTL architectures, it's possible that other, more advanced architectures or different parameter-sharing schemes could yield better results.
  • Future Directions: The authors suggest that future research should focus on increasing the amount of data available for ML in neuroimaging. They also hint at the potential of self-supervised or transfer learning approaches to overcome sample size limitations.

Summary updated

Abstract

There is a growing interest in using machine learning (ML) models to perform automatic diagnosis of psychiatric conditions; however, generalising the prediction of ML models to completely independent data can lead to sharp decrease in performance. Patients with different psychiatric diagnoses have traditionally been studied independently, yet there is a growing recognition of neuroimaging signatures shared across them as well as rare genetic copy number variants (CNVs). In this work, we assess the potential of multi-task learning (MTL) to improve accuracy by characterising multiple related conditions with a single model, making use of information shared across diagnostic categories and exposing the model to a larger and more diverse dataset. As a proof of concept, we first established the efficacy of MTL in a context where there is clearly information shared across tasks: the same target (age or sex) is predicted at different sites of data collection in a large functional magnetic resonance imaging (fMRI) dataset compiled from multiple studies. MTL generally led to substantial gains relative to independent prediction at each site. Performing scaling experiments on the UK Biobank, we observed that performance was highly dependent on sample size: for large sample sizes (N > 6000) sex prediction was better using MTL across three sites (N = K per site) than prediction at a single site (N = 3K), but for small samples (N < 500) MTL was actually detrimental for age prediction. We then used established machine-learning methods to benchmark the diagnostic accuracy of each of the 7 CNVs (N = 19-103) and 4 psychiatric conditions (N = 44-472) independently, replicating the accuracy previously reported in the literature on psychiatric conditions. We observed that MTL hurt performance when applied across the full set of diagnoses, and complementary analyses failed to identify pairs of conditions which would benefit from MTL. Taken together, our results show that if a successful multi-task diagnostic model of psychiatric conditions were to be developed with resting-state fMRI, it would likely require datasets with thousands of patients across different diagnoses.
BibTeX
@inproceedings{harvey2024challenges,
  author = {Annabelle Harvey and Clara Moreau and Kuldeep Kumar and Guillaume Huguet and Sebastian Urchs and Hanad Sharmarke and Khadijé Jizi and Charles-Olivier Martin and Nadine Younis and Petra Tamer and Martineau Jean‐Louis and Pierre Orban and Ana Isabel Silva and Jeremy Hall and Marianne van den Bree and Michael J. Owen and David E.J. Linden and Sarah Lippé and Carrie E. Bearden and Guillaume Dumas and Sébastien Jacquemont and Pierre Bellec},
  title = {{Challenges in multi-task learning for fMRI-based diagnosis: Benefits for psychiatric conditions and CNVs would likely require thousands of patients}},
  booktitle = {Imaging},
  year = {2024},
  doi = {10.1162/imag\_a\_00222},
}
Figure