Multimodal data fusion using sparse canonical correlation analysis and cooperative learning: a COVID-19 cohort study

Ahmet Gorkem Er, Daisy Yi Ding, Berrin Er, Mertcan Uzun, Mehmet Cakmak, Christoph Sadee, Gamze Durhan, Mustafa Nasuh Ozmen, Mine Durusu Tanriover, Arzu Topeli, Yesim Aydin Son, Robert Tibshirani, Serhat Unal, Olivier GevaertNpj Digit. Medicine 2024 · DOI

TL;DR

This paper tackles the challenge of integrating diverse data types (viral genomics, imaging, clinical, lab results) from COVID-19 patients to better understand disease progression and predict outcomes. They introduce a novel multimodal data fusion framework using sparse canonical correlation analysis (CCA) and cooperative learning, demonstrating its effectiveness in identifying distinct patient phenotypes and accurately predicting ICU admission.

What Problem Does This Paper Address?

The COVID-19 pandemic highlighted the complexity of the disease, with patient trajectories varying significantly. Understanding these variations is crucial for effective patient management, prognostication, and triage. This variability stems from a combination of host factors (age, comorbidities) and viral factors (mutations, variants). Analyzing these factors in isolation provides an incomplete picture. The challenge lies in effectively integrating high-dimensional, multi-scale biomedical data from various sources – including viral genome sequences, CT scans (radiomics), clinical records, and laboratory results – to uncover complex relationships and build robust predictive models. Existing approaches often struggle with the heterogeneity and high dimensionality of such multimodal datasets.

Methods

The study employs a two-pronged approach:

  • Unsupervised Data Fusion (Sparse CCA): Sparse Canonical Correlation Analysis (CCA) is used to identify and quantify relationships between pairs of different data modalities. The "sparse" aspect means it encourages simpler models by selecting only the most relevant features, making the relationships more interpretable. This helps in understanding how different data types correlate without pre-defined outcomes.
  • Supervised Prediction (Cooperative Learning): Cooperative learning is then used to build predictive models for ICU admission. This method allows for the integration of multiple data modalities, encouraging them to "cooperate" by aligning their predictions. It's a form of multi-view learning that can handle diverse data types and learn from their combined insights.

A key innovation is the use of Word2Vec encoding for viral genome sequencing data. Instead of traditional binary encoding of mutations, Word2Vec, a natural language processing technique, is used to create dense vector representations of viral strains. This captures semantic relationships between mutations and preserves phylogenetic information, potentially offering a richer representation of viral evolution and its impact.

Datasets

The study utilized data from a prospective cohort of 149 adult COVID-19 patients admitted to a tertiary care academic center. The data included:

  • Viral Genome Sequencing Data: From isolated SARS-CoV-2 strains.
  • Imaging Data: Primarily CT scans, from which radiomics features were extracted.
  • Clinical Data: Including patient demographics, comorbidities, and clinical scores.
  • Laboratory Results: Various blood and biochemical markers.

The viral genome data was sourced from the GISAID database.

Key Insights and Ideas

  • Word2Vec for Viral Genomics is Powerful: Using Word2Vec to encode viral genomes not only separates major SARS-CoV-2 variants but also preserves their phylogenetic relationships. This is a significant advancement over simpler binary encoding, offering a more nuanced understanding of viral evolution.
  • Multimodal Data Reveals Distinct Phenotypes: Sparse CCA successfully identified distinct clinical phenotypes by integrating laboratory results and clinical data. For example, one phenotype was characterized by high creatinine and low albumin, while another involved elderly, multi-morbid patients with renal disease.
  • Radiomics and Lab Results are Intertwined: Serum biomarkers related to severe disease and acute phase response showed significant correlation with specific radiomics features (especially in the LLL frequency channel). This suggests that imaging can capture systemic inflammatory responses.
  • Cooperative Learning Outperforms Unimodal Models: The supervised cooperative learning model, when integrating all four data modalities (clinical, lab, radiomics, and viral genomics with Word2Vec encoding), achieved superior prediction accuracy for ICU admission compared to models using single modalities.

Key Results

  • ICU Prediction Accuracy: The quadruple model combining clinical, laboratory, radiomics, and viral genome sequencing data (using Word2Vec encoding) achieved a mean Area Under the Curve (AUC) of 0.87 and an accuracy of 0.77 for predicting ICU admission. This significantly outperformed unimodal models.
  • Radiomics-Lab Correlation: A correlation of 0.596 (p < 0.001) was found between laboratory results and radiomics features in the LLL frequency channel.
  • Viral Variant Separation: Word2Vec encoding effectively separated major SARS-CoV-2 variants in a 2D embedding plot.
  • Clinical Phenotype Identification: Sparse CCA identified specific patient phenotypes based on combinations of laboratory and clinical data.

Limitations and Future Work

  • Small Sample Size: The authors acknowledge that the relatively small sample size might lead to overfitting. They mitigated this with stratified nested cross-validation and sparsity penalties.
  • Early Pandemic Data: The study data was collected early in the pandemic, meaning a limited number of patients were vaccinated. The effect of vaccination could not be properly quantified and was therefore ignored.
  • Future Directions: The paper suggests that future research should focus on further refining data fusion techniques and exploring the integration of viral genome sequencing into routine clinical patient management. The success of Word2Vec encoding hints at broader applications of NLP in analyzing biological sequences.

Summary updated

BibTeX
@article{er2024multimodal,
  author = {Ahmet Gorkem Er and Daisy Yi Ding and Berrin Er and Mertcan Uzun and Mehmet Cakmak and Christoph Sadee and Gamze Durhan and Mustafa Nasuh Ozmen and Mine Durusu Tanriover and Arzu Topeli and Yesim Aydin Son and Robert Tibshirani and Serhat Unal and Olivier Gevaert},
  title = {{Multimodal data fusion using sparse canonical correlation analysis and cooperative learning: a COVID-19 cohort study}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01128-2},
}
Figure