Abstract
Learning holistic computational representations in physical, chemical or biological systems requires the ability to process information from different distributions and modalities within the same model. Thus, the demand for multimodal machine learning models has sharply risen for modalities that go beyond vision and language, such as sequences, graphs, time series, or tabular data. While there are many available multimodal fusion and alignment approaches, most of them require end-to-end training, scale quadratically with the number of modalities, cannot handle cases of high modality imbalance in the training set, or are highly topology-specific, making them too restrictive for many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego), a general-purpose fusion framework to turn any set of encoders into a competitive multimodal model with no or minimal fine-tuning. We achieve this by introducing a wrapper for any unimodal encoder that enforces shape consistency between modality representations. It harmonises these representations by learning features in the frequency domain to enable model merging with little signal interference. We show that MM-Lego 1) can be used as a model merging method which achieves competitive performance with end-to-end fusion models without any fine-tuning, 2) can operate on any unimodal encoder, and 3) is a model fusion method that, with minimal fine-tuning, surpasses all benchmarks in five out of seven datasets.
Introduction
In the biomedical domain, patient data is inherently multimodal, spanning whole-slide pathology images, genomic tables, and longitudinal time-series data. Integrating these diverse sources is essential for accurate clinical prediction, yet existing fusion methods face significant hurdles. Most require expensive end-to-end training, scale poorly as the number of modalities increases, and are often restricted to specific model architectures. Furthermore, they typically require "paired" data—where every patient has every modality available—which is rarely the case in real-world clinical datasets.
We present Multimodal Lego (MM-Lego), a framework that allows researchers to modularly integrate specialized, pre-trained encoders. By standardizing representations and processing them in the frequency domain, MM-Lego enables high-performance multimodal modeling with minimal computational overhead.
Methodology: The MM-Lego Framework
MM-Lego introduces a modular approach to multimodal integration, categorized into three distinct stages: wrapping, merging, and fusing.
1. The LegoBlock Wrapper
The foundation of the framework is the LegoBlock, a universal wrapper for any unimodal encoder (e.g., ResNet for images, MLPs for tabular data). The LegoBlock enforces a consistent "latent shape" across different modalities, making them instantly compatible.
Crucially, the LegoBlock transforms these representations into the frequency domain using a Fast Fourier Transform (FFT). In standard spatial domains, combining signals often leads to destructive interference. In the frequency domain, different features occupy distinct "bins," allowing them to be merged with minimal signal loss. This mechanism uses an iterative update process that ensures the framework scales linearly ($O(M)$) with the number of modalities.
2. LegoMerge: Zero-Shot Integration
LegoMerge enables "zero-shot" model merging without any additional training. By taking the harmonic mean of frequency magnitudes and the arithmetic mean of phases, MM-Lego can combine models even if they were trained on completely different patient cohorts. This provides a performant multimodal model "for free" in data-constrained environments.
3. LegoFuse: Minimal Fine-Tuning
For tasks where limited training data is available, LegoFuse allows for rapid adaptation by "stacking" or "weaving" LegoBlocks. This process requires as few as two epochs of fine-tuning to achieve state-of-the-art results, facilitating cross-modal contextualization while maintaining the efficiency of the underlying pre-trained encoders.
Experimental Evaluation
The framework was evaluated on seven multimodal medical datasets, including TCGA (pathology and multi-omics), MIMIC-III (clinical tables and time-series), and SIIM-ISIC (skin cancer images and metadata).
Key Results
- State-of-the-Art Performance: LegoFuse outperformed all existing benchmarks in five out of seven datasets.
- Competitive Zero-Shot Merging: LegoMerge achieved performance comparable to complex end-to-end models without any fine-tuning.
- Computational Efficiency: MM-Lego avoids the quadratic complexity wall of traditional fusion. On the TCGA-UCEC dataset, it was over 10 times faster to train than state-of-the-art baselines like HEALNet (16.2s vs. 181.8s).
- Robustness to Unpaired Data: The framework successfully integrated models trained on disparate datasets, a critical requirement for practical biomedical applications.
Conclusion
MM-Lego provides a flexible, topology-agnostic framework for multimodal AI in medicine. By leveraging frequency-domain representations, it overcomes the traditional trade-offs between model complexity and integration ease. Future research will explore making the cross-attention adapters even more parameter-efficient and extending the framework to self-supervised learning paradigms.
Summary updated