Developing Multimodal Healthcare Foundation Model: From Data-driven to Knowledge-enhanced

Xiaochen WangAAAI 2025 · DOI

Abstract

While general-purpose foundation models have demonstrated remarkable capabilities, their application in healthcare is hindered by the domain's knowledge-intensive nature and strict privacy requirements. This paper proposes a paradigm shift from purely data-driven models to knowledge-enhanced systems. By integrating multimodal knowledge graphs with reinforcement learning techniques, specifically Direct Preference Optimization (DPO), we aim to create healthcare AI that is more factual, explainable, and robust against hallucinations.

Introduction

The success of Large Language Models (LLMs) like GPT-4 has sparked significant interest in developing specialized foundation models for healthcare. However, the medical domain presents unique challenges that "shoveling" massive amounts of general data into a model cannot solve. Medical data is inherently knowledge-intensive; interpreting a single patient record requires a deep understanding of clinical context, established taxonomies, and cross-modal correlations between text and imagery.

This research addresses the limitations of current data-driven approaches by proposing a framework that explicitly incorporates structured medical knowledge into the learning process. The goal is to move beyond implicit knowledge stored in model weights toward an explicit, verifiable reasoning architecture.

The Limitations of Data-Driven Approaches

Most current healthcare models rely on large-scale pretraining on unstructured datasets. This approach leads to three critical bottlenecks:

  1. Hallucinations: Models often generate medically incorrect information that sounds plausible, posing significant risks in clinical settings.
  2. Lack of Explainability: The "black box" nature of these models makes it difficult for clinicians to verify the reasoning behind a specific diagnosis or recommendation.
  3. Data Scarcity and Privacy: Regulations such as HIPAA limit the availability of large-scale, centralized datasets. Purely data-driven models struggle when high-quality, diverse data is siloed across different institutions.

Methodology: A Two-Stage Framework

To overcome these hurdles, we propose a two-stage project designed to bridge the gap between unstructured data and structured medical expertise.

Stage 1: Multimodal Knowledge Graph (KG) Construction

The first stage involves building a robust, multimodal knowledge graph. This graph uses the Unified Medical Language System (UMLS) as its foundational taxonomy. A key innovation in this stage is the integration of MIMIC-CXR data, where medical images (such as chest X-rays) are treated as "cross-modal instances." By linking visual data directly to structured medical concepts, the graph creates a bridge between pixel-level information and clinical terminology.

Stage 2: Knowledge Injection via Reinforcement Learning

Rather than traditional fine-tuning, we utilize Direct Preference Optimization (DPO) to align model outputs with the knowledge graph. In this framework, the KG serves as the "ground truth." The model is trained to prefer responses that are consistent with the structured facts in the KG. This effectively penalizes the model for hallucinations and steers it toward factual accuracy by using the graph as an automated, objective rater.

Key Innovations

  • Explicit Knowledge Representation: By moving from implicit weights to explicit graphs, the model's knowledge becomes verifiable by human experts.
  • Images as Knowledge Nodes: This approach treats images not just as raw input but as entities within a semantic network, allowing the model to reason about an X-ray in the context of established medical definitions.
  • DPO for Factuality: While DPO is traditionally used for human preference alignment (making models more helpful or polite), we repurpose it as a tool for factual alignment, ensuring the model adheres to clinical reality.

Preliminary Results and Previous Work

This research builds on several successful preliminary studies:

  • Federated Knowledge Injection (FEDMEKI): We developed a benchmark for injecting knowledge into models in a decentralized manner, successfully addressing privacy concerns by allowing models to learn from siloed data.
  • Multimodal EHR Assimilation: Previous work demonstrated that models could concurrently process heterogeneous healthcare data by exploiting correlations across different hierarchical levels within Electronic Health Records (EHRs).
  • Operational Efficiency: By utilizing Mix-of-Experts (MoE) and Parameter-efficient Fine-tuning (PEFT), we have shown that foundation models can handle diverse healthcare tasks without requiring prohibitive computational overhead.

Future Directions

Despite these advancements, several challenges remain. Future work will focus on Expert Verification, ensuring that the parameterized knowledge within the model remains aligned with evolving medical standards. Additionally, we are refining the KG-DPO pipeline to better manage Concept Ambiguity, where medical terms may have different meanings depending on the clinical context. Finally, we continue to explore decentralized learning to ensure that the next generation of healthcare AI can be trained safely on distributed, private data sources.

Summary updated

BibTeX
@inproceedings{wang2025developing,
  author = {Xiaochen Wang},
  title = {{Developing Multimodal Healthcare Foundation Model: From Data-driven to Knowledge-enhanced}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35230},
}
Figure