HEALNet - Hybrid Multi-Modal Fusion for Heterogeneous Biomedical Data

Konstantin Hemker, Nikola Simidjievski, Mateja Jamnik

Abstract

HEALNet (Hybrid Early-fusion Attention Learning Network) addresses the challenge of fusing structurally diverse biomedical data, such as high-resolution histopathology images and tabular genomic data. By employing an iterative attention mechanism with a shared latent bottleneck and modality-specific weights, HEALNet preserves data structure while capturing complex cross-modal interactions. Evaluated on TCGA datasets for cancer survival prediction, HEALNet achieves state-of-the-art performance, demonstrates robustness to missing data, and offers inherent interpretability through attention mapping.

Introduction

In the biomedical field, "multimodal" data is the standard. A single patient profile often includes 2D/3D radiology scans, gigapixel histopathology slides, and high-dimensional genomic (omic) data. Fusing these disparate sources is difficult because they lack a common semantic space. Existing strategies typically fall into three categories: early fusion (concatenating raw data, which destroys internal structure), late fusion (averaging separate model outputs, which misses cross-modal interactions), or intermediate fusion (learning embeddings first, which often results in "black box" models that struggle with missing data).

HEALNet bridges these gaps with a "hybrid early-fusion" approach designed to preserve data structure, capture complex interactions, and handle missing data gracefully in clinical settings.

Methodology

The core of HEALNet is an iterative attention architecture inspired by the Perceiver. It maintains a small, learned "latent array" ($S$) that acts as a shared memory space. In each layer, this latent array "queries" the raw input data of a specific modality.

HEALNet Architecture and Fusion Layer Detail

Crucially, HEALNet uses modality-specific attention weights, allowing the model to respect the unique structural properties of different data types (e.g., images vs. gene expression tables). The latent array passes through multiple fusion layers, picking up information sequentially to learn tacit biological interactions. Furthermore, because the architecture is iterative, missing modalities at inference time can be handled by simply skipping the corresponding update step, avoiding the need for noisy data imputation.

Experimental Setup

The researchers evaluated HEALNet primarily on survival analysis tasks using data from The Cancer Genome Atlas (TCGA) across four cancer types: BLCA (Bladder), BRCA (Breast), KIRP (Kidney), and UCEC (Uterine). The modalities included Whole Slide Images (WSI) of H&E-stained tissue and tabular omic data (RNAseq, mutations, and copy number variations). Additionally, the model was tested on the MIMIC-III dataset for mortality and disease classification to demonstrate its flexibility with time-series and tabular records.

Results and Discussion

HEALNet achieved state-of-the-art performance across all four TCGA datasets, outperforming established multimodal baselines such as Porpoise and MCAT in terms of the Concordance Index (c-Index).

Performance comparison (c-Index) across TCGA datasets

Key findings from the evaluation include:

  • Multimodal Uplift: HEALNet achieved up to a 7% improvement over the best unimodal benchmarks and a 4.5% improvement over existing fusion methods.
  • Structural Preservation: By using modality-specific weights, HEALNet outperforms baselines that flatten data too early, effectively treating images as images and tables as tables.
  • Efficiency: The latent bottleneck reduces dimensionality while retaining salient signals, keeping model complexity linear with respect to the number of modalities ($O(mn)$).
  • Robustness to Missing Data: In scenarios where 50% of modalities were missing at random during testing, HEALNet maintained stable performance, significantly outperforming late-fusion gating strategies.
  • Explainability: The model's direct attention to input data allows for the visualization of attention maps. Qualitative analysis confirmed that high-attention regions in WSIs often correspond to epithelial cell concentrations, a known marker for cancer origins.

Conclusion

HEALNet represents a significant step forward in multimodal biomedical data fusion. While it remains sensitive to hyperparameter tuning in "small $n$, large $p$" scenarios and currently relies on pre-trained feature extractors for imaging patches, its robust architecture offers a scalable and interpretable solution for clinical decision support. Future work will explore extending the framework to 3D radiology scans and longitudinal electronic health records.

Summary updated

BibTeX
@inproceedings{hemker2024healnet,
  author = {Konstantin Hemker and Nikola Simidjievski and Mateja Jamnik},
  title = {{HEALNet - Hybrid Multi-Modal Fusion for Heterogeneous Biomedical Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=HUxtJcQpDS},
}
Figure