Guiding Mixture-of-Experts with Temporal Multimodal Interactions

Xing Han, Hsing-Huan Chung, Joydeep Ghosh, Paul Pu Liang, Suchi Saria

Abstract

Massively multimodal learning involves processing hundreds of heterogeneous sensor streams. Traditional Mixture-of-Experts (MoE) models often fail to capture complex temporal interactions between these modalities. We present MERGE (Massively-multimodal Expert Routing for Generalized Exchange), a framework that leverages information theory to guide data routing. By quantifying time-delayed Redundancy, Uniqueness, and Synergy (RUS), MERGE assigns data to specialized experts, enhancing both performance and interpretability across healthcare and activity recognition tasks.

Introduction

Real-world applications, particularly in healthcare and wearable sensing, are "massively multimodal," involving hundreds of sensors like ECGs, lab results, and clinical notes. These streams exhibit varying sampling rates and significant temporal dependencies. While Mixture-of-Experts (MoE) architectures offer a scalable solution for such high-dimensional data, standard routing mechanisms typically rely on simple token similarity. This approach ignores the critical temporal dynamics and delayed cross-modal effects—such as a physiological change manifesting hours after a specific clinical event. Consequently, standard MoEs often fail to develop true expert specialization, limiting their effectiveness in complex, time-sensitive domains.

Methods: The MERGE Framework

MERGE addresses these limitations by incorporating information-theoretic signals into the routing process. The framework consists of two primary components: a Temporal RUS Quantifier and an Interaction-Aware Router.

Temporal RUS Quantifier

The model employs Partial Information Decomposition (PID) to calculate three distinct types of interactions between modalities over varying time lags ($\tau$):

  • Redundancy (R): Information shared across multiple modalities.
  • Uniqueness (U): Information specific to a single modality.
  • Synergy (S): Information that only becomes apparent when modalities are integrated.

To compute these values efficiently across multiple time scales, the authors developed a multi-scale BATCH estimator, providing a significant speedup over traditional step-wise estimation methods.

Interaction-Aware Router

Unlike standard routers, the MERGE router uses the calculated RUS values to direct tokens to specialized expert pools.

  • Redundancy Experts: Optimized for processing overlapping signals.
  • Uniqueness Experts: Focused on extracting modality-specific features.
  • Synergy Experts: Utilize cross-attention mechanisms to capture emergent patterns between disparate data streams.
Figure 1: Overview of the MERGE architecture, illustrating the modality encoders, the RUS-aware router, and the specialized expert pools (Uniqueness, Redundancy, and Synergy).

Experimental Evaluation

The framework was evaluated across six diverse benchmarks spanning healthcare (MIMIC-IV), activity recognition (PAMAP2, Opportunity), and affective computing (CMU-MOSI, WESAD).

Key Results

MERGE consistently outperformed state-of-the-art baselines, including standard Transformers and existing multimodal MoEs like FuseMoE.

  • Activity Recognition: On the PAMAP2 dataset, MERGE achieved an accuracy of 91.37%, compared to 87.74% for FuseMoE.
  • Healthcare Robustness: In predicting in-hospital mortality on MIMIC-IV, MERGE reached an AUROC of 85.40, surpassing specialized fusion models like MISTS and MulT.

Temporal Dynamics and Interpretability

A core finding of this work is that multimodal interactions are rarely instantaneous. As shown in Figure 6, the relative importance of redundancy, uniqueness, and synergy shifts significantly across different time lags. By explicitly modeling these shifts, MERGE not only improves accuracy but also provides a layer of interpretability, allowing researchers to observe which types of interactions the model prioritizes for specific predictions.

Figure 6: Temporal RUS profiles across different datasets, showing how Redundancy (R), Uniqueness (U), and Synergy (S) evolve over various time lags.

Conclusion

MERGE demonstrates that information theory provides a principled and effective signal for guiding expert routing in massively multimodal settings. By accounting for temporal delays and complex cross-modal interactions, the framework achieves superior performance and enhanced transparency. Future work will explore extending this approach to continuous spatio-temporal dynamics and integrating interaction-aware routing into larger-scale multimodal foundation models.

Summary updated

BibTeX
@inproceedings{han2026guiding,
  author = {Xing Han and Hsing-Huan Chung and Joydeep Ghosh and Paul Pu Liang and Suchi Saria},
  title = {{Guiding Mixture-of-Experts with Temporal Multimodal Interactions}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=qF9WJxvHX8},
}
Figure