Abstract
One important challenge of applying deep learning to electronic health records (EHR) is the complexity of their multimodal structure. EHR usually contains a mixture of structured (codes) and unstructured (free-text) data with sparse and irregular longitudinal features -- all of which doctors utilize when making decisions. In the deep learning regime, determining how different modality representations should be fused together is a difficult problem, which is often addressed by handcrafted modeling and intuition. In this work, we extend state-of-the-art neural architecture search (NAS) methods and propose MUltimodal Fusion Architecture SeArch (MUFASA) to simultaneously search across multimodal fusion strategies and modality-specific architectures for the first time. We demonstrate empirically that our MUFASA method outperforms established unimodal NAS on public EHR data with comparable computation costs. In addition, MUFASA produces architectures that outperform Transformer and Evolved Transformer. Compared with these baselines on CCS diagnosis code prediction, our discovered models improve top-5 recall from 0.88 to 0.91 and demonstrate the ability to generalize to other EHR tasks. Studying our top architecture in depth, we provide empirical evidence that MUFASA's improvements are derived from its ability to both customize modeling for each modality and find effective fusion strategies.
The Complexity of Multimodal EHR Data
Electronic Health Records (EHR) are a goldmine for deep learning, but they are notoriously messy. A single patient's record is a "multimodal" mix of categorical codes (diagnoses and procedures), continuous values (heart rate, blood pressure, and lab results), and unstructured text (clinical notes written by doctors). These data points are often sparse and occur at irregular intervals over time, making integration difficult.
The central challenge lies in fusion: determining how to effectively combine these disparate data types. Most researchers rely on "early fusion" (concatenating all inputs at the start) or "late fusion" (averaging predictions at the end) based on intuition. However, different modalities have different structures and "rhythms." A model optimized for processing heart rate sequences might be poorly suited for clinical notes. Hand-designing custom architectures for every possible combination of medical data is a significant bottleneck that doesn't scale.
MUFASA: Automating Fusion Architecture Search
To address this, we propose MUFASA (MUltimodal Fusion Architecture SeArch), which extends evolutionary Neural Architecture Search (NAS) to the multimodal domain. Instead of searching for a single monolithic model, MUFASA searches for a collection of sub-architectures:
- Modality-Specific Architectures ($g_i$): Independent "lanes" that process each data type (e.g., one for labs, one for notes).
- Fusion Architecture ($h$): A "master" architecture that decides at which layer and through which operation (addition, concatenation, etc.) to integrate information from the modality lanes.
The search utilizes Tournament Selection, an evolutionary algorithm where a population of architectures "mutates" over time. The "fittest" models—those with the best validation performance—survive and pass on their structural "genes" to the next generation. This allows the system to discover complex hybrid fusion strategies that humans might never think to try.
Key Insights and Results
Our evaluation on the MIMIC-III dataset, which contains data from over 50,000 hospital admissions, yielded several key insights:
- Modality Customization is Essential: The search discovered that "siloing" modalities—processing them independently for several layers before fusion—is significantly more effective than early mixing.
- Specialized vs. Wide Models: While unimodal NAS tends to create "wide" models to view the same data from multiple angles, MUFASA creates "deep and specialized" models where each branch is tailored to the unique properties of its data type.
- Superior Performance: On the CCS diagnosis task, MUFASA achieved a top-5 recall of 0.9075, outperforming standard Transformers (0.8756) and traditional unimodal NAS (0.8841).
- Robust Generalization: The architecture discovered on the CCS task also performed best on the more granular ICD-9 task (0.3327 AUCPR), suggesting the discovered fusion logic is robust across different medical prediction problems.
Conclusion
MUFASA demonstrates that automating the design of both modality-specific architectures and their fusion strategies leads to significant performance gains in EHR tasks. By moving away from "one-size-fits-all" models, we can better capture the nuances of complex medical data. Future work will look to incorporate high-bandwidth modalities like medical imaging and continuous waveforms into the fusion search space.
Summary updated