Multimodal machine learning in precision health: A scoping review

Adrienne S. Kline, Hanyin Wang, Yikuan Li, Saya R. Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, Yuan LuoNpj Digit. Medicine 2022 · DOI

TL;DR

This paper presents a comprehensive scoping review of multimodal machine learning in precision health, identifying key trends, challenges, and future directions. It highlights that combining diverse data sources significantly improves predictive accuracy in healthcare, but also points to a lack of clear clinical deployment strategies and FDA-approved tools.

What Problem Does This Paper Address?

Precision health aims to tailor medical decisions, treatments, and practices to individual patient phenotypes. This requires integrating a vast array of data, from lab results and imaging to genetic information and clinical notes. While single-source (unimodal) machine learning has made strides, clinical decision-making often relies on synthesizing information from multiple sources. The challenge lies in effectively fusing these disparate data types to build robust and accurate predictive models that can truly inform personalized healthcare. This review addresses the need to map out the current landscape of multimodal machine learning in health, identify common practices, and pinpoint areas ripe for further research and development.

Methods

The authors conducted a scoping review following PRISMA extension guidelines. They systematically searched PubMed, Google Scholar, and IEEE Xplore for articles published between 2011 and 2021 that utilized multimodal machine learning for health applications. A total of 128 articles were included in the analysis. The review categorized studies based on the type of data fusion employed (early, intermediate, late), the machine learning techniques used, the health domains investigated, and the validation methods. They also analyzed coding platforms, publication trends, and author locations.

Datasets

The paper does not focus on specific datasets for training or evaluation but rather analyzes the types of data sources used across the reviewed literature. The most common data combinations were imaging and Electronic Health Records (EHR), followed by combinations involving EHR, genomics, and text data. The review highlights that the datasets often came from single sites and had limitations such as missing data and imbalanced samples.

Key Insights and Ideas

  • Multimodal Fusion Boosts Accuracy: The core insight is that combining multiple data sources consistently leads to improved predictive accuracy compared to unimodal approaches. Studies that compared multimodal to unimodal predictions showed an average increase of 6.4% in accuracy.
  • Neurology and Oncology Dominate: The most frequently studied health areas using multimodal machine learning were neurology (particularly Alzheimer's disease) and oncology. This is likely due to the availability of well-curated, multimodal datasets in these fields.
  • Early Fusion is Prevalent: Early fusion (data-level merging) was the most common strategy. This is attributed to its relative simplicity, requiring less computational complexity and fewer distinct models compared to intermediate or late fusion.
  • Clinical Translation is Lagging: A significant gap exists between research and clinical application. The review found a lack of clear clinical deployment strategies, FDA approval for developed tools, and insufficient analysis of how multimodal approaches might address healthcare disparities.
  • Interpretability Remains a Challenge: While multimodal models offer improved performance, they often suffer from reduced interpretability, making it harder for clinicians to trust and adopt them.

Key Results

  • 128 articles were included in the review.
  • Neurology and oncology were the most common health areas investigated.
  • Imaging and EHR data fusion was the most prevalent combination (52 studies).
  • An average increase of 6.4% in predictive accuracy was observed when comparing multimodal to unimodal approaches.
  • Early fusion was the most common data merging strategy.
  • Less than half (37.6%) of the papers were published in journals intended for a clinical audience.
  • None of the reviewed studies had developed tools with FDA approval for clinical use.

Limitations and Future Work

The authors acknowledge that this is a scoping review, not a systematic review, meaning some relevant papers might have been missed. They also note that a critical appraisal of the methodological quality of individual studies was not performed.

Future directions proposed include:

  • Exploring Undersaturated Health Topics: Investigating areas like dermatology, hematology, and medication/drug-related issues, which are currently underrepresented.
  • Improving Data Harmonization and Management: Developing better methods to handle missing data, create prospective studies, and expand datasets to include multiple sites to improve generalizability and reduce bias.
  • Developing Interpretable Models: Focusing on "white-box" methods or techniques that enhance model interpretability to build trust and facilitate clinical adoption.
  • Facilitating Clinical Translation: Encouraging more research on FDA approval pathways, developing user-friendly interfaces, and ensuring clinical relevance and actionability of the models.
  • Addressing Bias and Generalizability: Exploring multimodal data avenues to combat issues associated with geographic, gender, ethnic, and racial subpopulations.
  • Creating Open-Access Resources: Advocating for open-access pipelines and libraries to speed up data conversion and model development.

Summary updated

BibTeX
@article{kline2022multimodal,
  author = {Adrienne S. Kline and Hanyin Wang and Yikuan Li and Saya R. Dennis and Meghan Hutch and Zhenxing Xu and Fei Wang and Feixiong Cheng and Yuan Luo},
  title = {{Multimodal machine learning in precision health: A scoping review}},
  journal = {Npj Digit. Medicine},
  year = {2022},
  doi = {10.1038/s41746-022-00712-8},
}
Figure