Multimodal Learning for Scalable Representation of High-Dimensional Medical Data

Areej Alsaafin, Abubakr Shafique, Saghir Alfasly, Krishna R. Kalari, H. R. Tizhoosh

Abstract

Integrating artificial intelligence (AI) with healthcare data is rapidly transforming medical diagnostics and driving progress toward precision medicine. However, effectively leveraging multimodal data, particularly digital pathology whole slide images (WSIs) and genomic sequencing, remains a significant challenge due to the intrinsic heterogeneity of these modalities and the need for scalable and interpretable frameworks. Existing diagnostic models typically operate on unimodal data, overlooking critical cross-modal interactions that can yield richer clinical insights. We introduce MarbliX (Multimodal Association and Retrieval with Binary Latent Indexed matriX), a self-supervised framework that learns to embed WSIs and immunogenomic profiles into compact, scalable binary codes, termed "monograms." By optimizing a triplet contrastive objective across modalities, MarbliX captures high-resolution patient similarity in a unified latent space, enabling efficient retrieval of clinically relevant cases and facilitating case-based reasoning. In lung cancer, MarbliX achieves 85-89% across all evaluation metrics, outperforming histopathology (69-71%) and immunogenomics (73-76%). In kidney cancer, real-valued monograms yield the strongest performance (F1: 80-83%, Accuracy: 87-90%), with binary monograms slightly lower (F1: 78-82%).

Introduction

Modern oncology generates a vast amount of data, primarily from two sources: Whole Slide Images (WSIs) of tissue and genomic sequencing. While both contain vital clues for diagnosis and treatment, they are fundamentally different. Images are dense and spatial, while immunogenomic data (such as T-cell and B-cell receptor sequences) is sparse, high-dimensional, and structurally complex. Most existing AI models are unimodal, missing the rich cross-talk between a tumor’s morphological appearance and its genetic makeup. Furthermore, existing multimodal models often produce massive, real-valued embeddings that are difficult to store or search in large-scale hospital archives. MarbliX addresses the need for a framework that is both multimodal and scalable, using compact binary codes for fast retrieval.

Methods

The MarbliX (Multimodal Association and Retrieval with Binary Latent Indexed matriX) pipeline consists of three main stages:

  1. Unimodal Transformation: WSIs are condensed into representative patches using the SPLICE algorithm and embedded via a pre-trained DINO ViT. Simultaneously, TCR/BCR sequences are harmonized and embedded using a BERT-based model.
  2. Multimodal Latent Association: Hybrid autoencoders are employed where one modality is tasked with reconstructing the other (e.g., using image features to predict genomic features). This forces the model to ignore modality-specific noise, such as scanner artifacts, and focus on shared biological signals.
  3. Monogram Generation: The resulting latent vectors are combined via an outer product and passed through a network trained with triplet contrastive loss. This network learns to minimize the distance between patients with the same diagnosis while maximizing the distance between different ones. The final output is a 64-bit binary matrix (8x8) called a "monogram."
Overview of the MarbliX framework for multimodal medical data integration.

Because the monograms are binary, searching a database for similar patients becomes a simple matter of calculating Hamming distance using fast XOR operations, which is computationally trivial even for millions of records.

Results

The study utilized data from The Cancer Genome Atlas (TCGA), focusing on lung cancer (535 LUAD and 510 LUSC cases) and kidney cancer (508 KIRC, 248 KIRP, and 38 KICH cases).

In lung cancer, MarbliX achieved 85–89% across all evaluation metrics (F1, Accuracy, Precision, Recall), significantly outperforming histopathology alone (69–71%) and immunogenomics alone (73–76%). This demonstrates that the two modalities provide complementary, not just redundant, information. For kidney cancer, real-valued monograms performed best (87–90% accuracy), while the compressed binary monograms remained strong with an F1-score of 78–82%.

Qualitative analysis of the monograms showed that patients with the same cancer subtype produced consistent binary patterns, while different subtypes showed clear, distinct "bit-flips," confirming the monogram's discriminative power.

Discussion and Conclusion

MarbliX provides a scalable solution for integrating high-dimensional medical data. By forcing cross-modal reconstruction, the model effectively filters out non-biological visual noise like tissue folds or pen marks. Furthermore, the framework is highly accessible; once features are extracted, the association and triplet training can be performed on a standard CPU in about 15 minutes.

While the initial feature extraction from gigapixel images remains a preprocessing bottleneck, the resulting bimodal patient codes are more stable than unimodal ones. Perturbations in one modality, such as a blurry slide, are less likely to distort the entire representation. Future work will involve expanding this framework to other cancer types and integrating more diverse "omics" layers to further advance precision medicine.

Summary updated

Abstract

Integrating artificial intelligence (AI) with healthcare data is rapidly transforming medical diagnostics and driving progress toward precision medicine. However, effectively leveraging multimodal data, particularly digital pathology whole slide images (WSIs) and genomic sequencing, remains a significant challenge due to the intrinsic heterogeneity of these modalities and the need for scalable and interpretable frameworks. Existing diagnostic models typically operate on unimodal data, overlooking critical cross-modal interactions that can yield richer clinical insights. We introduce MarbliX (Multimodal Association and Retrieval with Binary Latent Indexed matriX), a self-supervised framework that learns to embed WSIs and immunogenomic profiles into compact, scalable binary codes, termed ``monogram.'' By optimizing a triplet contrastive objective across modalities, MarbliX captures high-resolution patient similarity in a unified latent space, enabling efficient retrieval of clinically relevant cases and facilitating case-based reasoning. \textcolor{black}{In lung cancer, MarbliX achieves 85-89\% across all evaluation metrics, outperforming histopathology (69-71\%) and immunogenomics (73-76\%). In kidney cancer, real-valued monograms yield the strongest performance (F1: 80-83\%, Accuracy: 87-90\%), with binary monograms slightly lower (F1: 78-82\%).
BibTeX
@misc{alsaafin2024multimodal,
  author = {Areej Alsaafin and Abubakr Shafique and Saghir Alfasly and Krishna R. Kalari and H. R. Tizhoosh},
  title = {{Multimodal Learning for Scalable Representation of High-Dimensional Medical Data}},
  year = {2024},
  eprint = {2409.13115},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.13115},
}
Figure