MM-Lego: Modular Biomedical Multimodal Models with Minimal Fine-Tuning

Konstantin Hemker, Nikola Simidjievski, Mateja Jamnik

Abstract

MM-Lego is a modular framework designed to integrate disparate pre-trained unimodal models—such as those for medical imaging, genomics, and time-series data—into high-performing multimodal systems. By leveraging frequency-domain harmonization, MM-Lego enables model fusion with zero or minimal fine-tuning, overcoming common challenges like signal interference and architectural incompatibility. The framework scales linearly with the number of modalities and achieves state-of-the-art performance on complex biomedical datasets with significantly reduced computational overhead.

Introduction: The Challenge of Biomedical Data Fusion

Biomedical research often involves heterogeneous data types for a single patient, including pathology slides, gene expression profiles, and longitudinal clinical records. Traditional multimodal fusion methods are typically monolithic, requiring all data modalities to be present for every sample and necessitating expensive end-to-end training. Furthermore, these methods often assume "topological equivalence," making it difficult to merge models with fundamentally different architectures, such as Convolutional Neural Networks (CNNs) and Multi-Layer Perceptrons (MLPs).

MM-Lego addresses these limitations by providing a flexible, "snap-together" architecture that allows researchers to combine independently trained models into a unified system without requiring paired data or identical model structures.

Methods: A Modular Approach to Fusion

The MM-Lego framework is built upon three primary components that facilitate seamless model integration.

Overview of the MM-Lego framework, illustrating the Wrap, Merge, and Fuse stages.

1. LegoBlock: The Universal Wrapper

The foundation of the framework is the LegoBlock, a wrapper that encapsulates any unimodal encoder. It maps diverse latent representations into a consistent shape and processes information in the frequency domain using Fast Fourier Transforms (FFT). This approach ensures that the internal architecture of the original encoder remains irrelevant to the fusion process.

Detailed view of the Fourier Lego Block, which learns modality-agnostic frequency representations.

2. LegoMerge: Zero-Shot Integration

LegoMerge enables the combination of multiple LegoBlocks with zero additional training. By merging frequency-domain representations—specifically using the harmonic mean of magnitudes and the arithmetic mean of phases—it prevents "signal interference," a phenomenon where features from different modalities cancel each other out in the spatial domain.

3. LegoFuse: Minimal Fine-Tuning

For scenarios where cross-modal interaction is critical, LegoFuse allows for minimal fine-tuning (as few as 2 epochs). It utilizes "stacking" or "weaving" techniques to pass latent states between blocks, significantly boosting performance with negligible computational cost.

Experimental Evaluation

The framework was rigorously tested on seven multimodal medical datasets from three major sources:

  • TCGA (The Cancer Genome Atlas): Four cancer subtypes (BLCA, BRCA, KIRP, UCEC) involving pathology images, gene expression, mutations, and copy number variations.
  • MIMIC-III: Intensive care data for mortality and disease classification, combining clinical variables and vital signs.
  • SIIM-ISIC: Skin cancer detection using clinical metadata and lesion images.

Key Insights and Results

  • Frequency Domain Advantages: Merging in the frequency domain preserves salient signals from disparate sources (e.g., images and tables) that might otherwise be lost to noise in the spatial domain.
  • Topology Agnosticism: The LegoBlock interface allows for the mixing of Transformers, CNNs, and MLPs without modification.
  • Handling Unpaired Data: MM-Lego remains effective even when modalities are trained on non-overlapping patient cohorts, a common constraint in medical data.
  • Efficiency and Scalability: MM-Lego scales linearly ($O(M)$) with the number of modalities. In practice, LegoFuse achieved an 11x speedup over baselines like HEALNet on the TCGA-UCEC dataset.
  • Superior Performance: LegoMerge (zero-shot) proved competitive with end-to-end models, while LegoFuse (minimal fine-tuning) surpassed state-of-the-art benchmarks on five of the seven datasets tested.

Limitations and Future Directions

While MM-Lego offers a powerful solution for biomedical fusion, there are areas for further development. The current cross-attention mechanism could be optimized for greater parameter efficiency. Additionally, while highly effective in the biomedical domain, the framework's performance on general vision-language tasks remains to be explored. Future work will also investigate self-supervised methods for fitting LegoBlock adapters to reduce dependency on supervised labels.

Summary updated

BibTeX
@inproceedings{hemker2025mm,
  author = {Konstantin Hemker and Nikola Simidjievski and Mateja Jamnik},
  title = {{MM-Lego: Modular Biomedical Multimodal Models with Minimal Fine-Tuning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=pH543jrbe8},
}
Figure