Abstract
Massively multimodal learning involves processing hundreds of heterogeneous sensor streams. Traditional Mixture-of-Experts (MoE) models often fail to capture complex temporal interactions between these modalities. We present MERGE (Massively-multimodal Expert Routing for Generalized Exchange), a framework that leverages information theory to guide data routing. By quantifying time-delayed Redundancy, Uniqueness, and Synergy (RUS), MERGE assigns data to specialized experts, enhancing both performance and interpretability across healthcare and activity recognition tasks.
Introduction
Real-world applications, particularly in healthcare and wearable sensing, are "massively multimodal," involving hundreds of sensors like ECGs, lab results, and clinical notes. These streams exhibit varying sampling rates and significant temporal dependencies. While Mixture-of-Experts (MoE) architectures offer a scalable solution for such high-dimensional data, standard routing mechanisms typically rely on simple token similarity. This approach ignores the critical temporal dynamics and delayed cross-modal effects—such as a physiological change manifesting hours after a specific clinical event. Consequently, standard MoEs often fail to develop true expert specialization, limiting their effectiveness in complex, time-sensitive domains.
Methods: The MERGE Framework
MERGE addresses these limitations by incorporating information-theoretic signals into the routing process. The framework consists of two primary components: a Temporal RUS Quantifier and an Interaction-Aware Router.
Temporal RUS Quantifier
The model employs Partial Information Decomposition (PID) to calculate three distinct types of interactions between modalities over varying time lags ($\tau$):
- Redundancy (R): Information shared across multiple modalities.
- Uniqueness (U): Information specific to a single modality.
- Synergy (S): Information that only becomes apparent when modalities are integrated.
To compute these values efficiently across multiple time scales, the authors developed a multi-scale BATCH estimator, providing a significant speedup over traditional step-wise estimation methods.
Interaction-Aware Router
Unlike standard routers, the MERGE router uses the calculated RUS values to direct tokens to specialized expert pools.
- Redundancy Experts: Optimized for processing overlapping signals.
- Uniqueness Experts: Focused on extracting modality-specific features.
- Synergy Experts: Utilize cross-attention mechanisms to capture emergent patterns between disparate data streams.
Experimental Evaluation
The framework was evaluated across six diverse benchmarks spanning healthcare (MIMIC-IV), activity recognition (PAMAP2, Opportunity), and affective computing (CMU-MOSI, WESAD).
Key Results
MERGE consistently outperformed state-of-the-art baselines, including standard Transformers and existing multimodal MoEs like FuseMoE.
- Activity Recognition: On the PAMAP2 dataset, MERGE achieved an accuracy of 91.37%, compared to 87.74% for FuseMoE.
- Healthcare Robustness: In predicting in-hospital mortality on MIMIC-IV, MERGE reached an AUROC of 85.40, surpassing specialized fusion models like MISTS and MulT.
Temporal Dynamics and Interpretability
A core finding of this work is that multimodal interactions are rarely instantaneous. As shown in Figure 6, the relative importance of redundancy, uniqueness, and synergy shifts significantly across different time lags. By explicitly modeling these shifts, MERGE not only improves accuracy but also provides a layer of interpretability, allowing researchers to observe which types of interactions the model prioritizes for specific predictions.
Conclusion
MERGE demonstrates that information theory provides a principled and effective signal for guiding expert routing in massively multimodal settings. By accounting for temporal delays and complex cross-modal interactions, the framework achieves superior performance and enhanced transparency. Future work will explore extending this approach to continuous spatio-temporal dynamics and integrating interaction-aware routing into larger-scale multimodal foundation models.
Summary updated