Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine

Konstantin Hemker, Nikola Simidjievski, Mateja Jamnik

Introduction

Learning holistic computational representations in complex physical or biological systems requires processing information from diverse distributions and modalities. Multimodal Lego (MM-Lego) is a modular framework designed to transform independent unimodal encoders—such as those for images, tabular data, or time series—into a unified multimodal model with little to no extra training. By mapping different data types into a shared frequency-domain latent space, MM-Lego achieves state-of-the-art performance while solving the "missing data" and scalability headaches common in medical research.

The Challenge of Biomedical Data

In biomedicine, clinical data is rarely uniform. A single patient might have a pathology slide (image), gene expression levels (tabular), and heart rate monitors (time series). However, most current AI models struggle to integrate these diverse inputs due to several critical bottlenecks:

  1. End-to-End Costs: Training a multimodal model from scratch on all modalities simultaneously is computationally expensive.
  2. The "Paired Data" Trap: Many traditional fusion methods require every patient to have every type of data in the training set, which is a rarity in real-world clinical practice.
  3. Scalability: Standard fusion techniques, such as cross-attention, often scale quadratically ($O(M^2)$) as more modalities ($M$) are added, making them prohibitively slow.
  4. Topology Mismatch: It is inherently difficult to merge a model designed for spatial images with one designed for flat spreadsheets because their internal architectures are fundamentally different.

The MM-Lego Framework

To overcome these limitations, the authors introduce the LegoBlock, a versatile wrapper that can be placed around any unimodal encoder.

Overview of the Multimodal Lego (MM-Lego) framework, illustrating the unimodal wrapper and the two fusion strategies: zero-shot merging and minimal-tuning fusion.

The framework operates on a few core principles:

  • Shape Consistency: The LegoBlock forces different encoders to output a standardized latent representation (a $c \times d$ matrix), regardless of whether the original input was a 2D image or a 1D sequence.
  • The Frequency Domain Trick: Instead of processing data in the standard "spatial" domain, LegoBlocks utilize the Fast Fourier Transform (FFT) to learn features in the frequency domain. Because frequency components are orthogonal, they are far less likely to cancel each other out when combined, effectively mitigating signal interference.

Once the unimodal encoders are wrapped, MM-Lego offers two distinct strategies for combining them:

  • LegoMerge (Zero-Shot): A method where pre-trained unimodal blocks are combined using the harmonic means of their magnitudes and the arithmetic means of their phases. This requires absolutely no multimodal training.
  • LegoFuse (Minimal-Tuning): A strategy where the blocks are stacked or "weaved" together and fine-tuned for just a few epochs (typically two) to allow the modalities to contextualize each other.

Experimental Setup

The framework was rigorously evaluated on seven multimodal medical datasets across three major sources:

  • TCGA (The Cancer Genome Atlas): Covering four cancer types (BLCA, BRCA, KIRP, UCEC) and involving pathology images, gene mutations, and RNA sequencing for survival analysis.
  • MIMIC-III: Intensive care data combining clinical tabular variables and time-series vital signs for mortality and disease classification.
  • ISIC: Skin lesion images paired with clinical metadata for melanoma prediction.

Key Insights and Results

Frequency Domain as a "Universal Language"

By moving to the frequency domain, the model creates a harmonized latent space where different data types can be merged mathematically without destroying the underlying signal. The authors discovered that using the harmonic mean to merge magnitudes in this domain is highly robust to outliers, preventing any single modality from dominating the others.

Robustness to Unpaired Data

Perhaps the most impressive feature of MM-Lego is its ability to function even if the individual encoders were trained on completely different sets of patients. Researchers can effectively "lego-merge" a model trained on Patient Group A's images with a model trained on Patient Group B's genomics.

Performance and Efficiency

  • State-of-the-Art (SOTA): With only 2 epochs of fine-tuning, LegoFuse outperformed all benchmarks in 5 out of 7 datasets.
  • Zero-Shot Success: LegoMerge achieved competitive performance with complex end-to-end models without any fine-tuning, significantly outperforming simple ensembles. It provided a performance uplift of up to 5.7% over the best unimodal model without seeing a single multimodal training sample.
  • Speed and Scaling: MM-Lego scales linearly ($O(M)$) with the number of modalities. On the TCGA-UCEC dataset, LegoFuse was nearly 11x faster to train than the domain-specific HEALNet baseline (16.2s vs 181.8s total wall time).

Limitations and Future Work

While highly effective, the framework does have limitations. The LegoBlock adapters currently require some initial supervised fitting to the specific encoder, though future work could explore self-supervised methods for this step. Additionally, the fusion layer within the LegoBlock could be further optimized for parameter efficiency. Finally, while the authors believe the framework is general-purpose, its current evaluation is strictly bounded to biomedical data; its efficacy on standard vision-language tasks (like CLIP-style models) remains an open question for future research.

Summary updated

Abstract

Learning holistic computational representations in physical, chemical or biological systems requires the ability to process information from different distributions and modalities within the same model. Thus, the demand for multimodal machine learning models has sharply risen for modalities that go beyond vision and language, such as sequences, graphs, time series, or tabular data. While there are many available multimodal fusion and alignment approaches, most of them require end-to-end training, scale quadratically with the number of modalities, cannot handle cases of high modality imbalance in the training set, or are highly topology-specific, making them too restrictive for many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego), a general-purpose fusion framework to turn any set of encoders into a competitive multimodal model with no or minimal fine-tuning. We achieve this by introducing a wrapper for any unimodal encoder that enforces shape consistency between modality representations. It harmonises these representations by learning features in the frequency domain to enable model merging with little signal interference. We show that MM-Lego 1) can be used as a model merging method which achieves competitive performance with end-to-end fusion models without any fine-tuning, 2) can operate on any unimodal encoder, and 3) is a model fusion method that, with minimal fine-tuning, surpasses all benchmarks in five out of seven datasets.
BibTeX
@inproceedings{hemker2025multimodal,
  author = {Konstantin Hemker and Nikola Simidjievski and Mateja Jamnik},
  title = {{Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=pH543jrbe8},
}
Figure