FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, Suchi Saria

TL;DR

FuseMoE is a novel Mixture-of-Experts (MoE) framework designed to handle "FlexiModal" data—multimodal inputs characterized by irregular temporal sampling and frequent missing values. By introducing a new Laplace gating function, the authors provide both a theoretical guarantee of faster convergence and empirical proof of superior performance across clinical, sentiment, and activity recognition tasks.

What Problem Does This Paper Address?

In high-stakes fields like healthcare, machine learning models rarely encounter "clean" data. Instead, they face FlexiModal Data: a messy mix of modalities (e.g., clinical notes, vital signs, X-rays) where samples are collected at irregular intervals and many modalities are simply missing for certain individuals.

Existing multimodal fusion techniques often fall short because they:

  1. Lack Scalability: Many are designed for only two modalities or rely on computationally expensive pairwise comparisons.
  2. Struggle with Missingness: They either require all modalities to be present or rely on suboptimal data imputation.
  3. Ignore Irregularity: They often fail to account for the varying temporal dynamics inherent in real-world time-series data.

Methods

FuseMoE tackles these challenges by combining a specialized encoder with a sparse MoE backbone.

  • Modality and Irregularity Encoder: The model uses a discretized multi-time attention (mTAND) module to map irregularly sampled observations into a standardized, regularly sampled embedding space.
  • Sparse MoE Backbone: Instead of a single dense layer, FuseMoE uses a set of Feed-Forward Network (FFN) "experts." A router selects the best experts for a given input, allowing the model to scale capacity without a massive increase in computational cost.
  • The Laplace Gating Function: This is the paper's primary innovation. Traditional MoE models use a Softmax gating function based on the inner product of inputs and expert weights. FuseMoE uses a Laplace gating function based on $L_2$ (Euclidean) distance: $h_l(x) = \text{Top K}(-\|W - x\|_2)$.
  • Flexible Routing Strategies: The authors explore three router designs to optimize how different modalities interact with experts:
  • Joint Router: All modalities are concatenated and sent to a shared pool of experts.
  • Per-modality Routers: Each modality has its own router but shares a common pool of experts.
  • Disjoint Experts & Routers: Each modality has its own router and its own dedicated pool of specialized experts.
Figure 2: We present three exemplary designs of the Top-K router for effective multimodal fusion, considering an input scenario with three modalities: Time-Series (TS), Text (TXT), and images (IMG). (a) The joint router design utilizes a concatenated embedding of all modalities, directing this combined input to selected experts. (b) In the modality-specific router design, each modality’s embedding is independently assigned to a shared pool of experts. (c) The third design variant also uses modality-specific routers but assigns each modality’s embedding to separate pools of experts, each pool uniquely tailored to process a specific modality type.

Datasets

The authors evaluated FuseMoE on a diverse range of benchmarks to prove its versatility:

  • Clinical (MIMIC-III & MIMIC-IV): Electronic Health Records containing vitals, notes, X-rays, and ECGs. Tasks included mortality prediction (48-IHM), length-of-stay (LOS), and phenotype classification (25-PHE).
  • Sentiment Analysis (CMU-MOSI & MOSEI): Multimodal data consisting of video, audio, and text.
  • Activity Recognition (PAM): Data from 17 sensors monitoring daily living activities.
  • Image Classification (CIFAR-10): Used specifically to validate the Laplace gating function within a Vision-MoE framework.

Key Insights and Ideas

  • Balanced Expert Selection: Softmax gating can lead to "representation collapse," where a few experts dominate because the inner product is sensitive to magnitude. Laplace gating, by using Euclidean distance, ensures a more balanced selection process, which is crucial when dealing with heterogeneous multimodal features.
  • Handling Missing Data Dynamically: By using per-modality routers and a learnable "missing indicator" embedding, FuseMoE can dynamically down-weight the influence of experts responsible for absent data without needing explicit imputation.
  • Theoretical Convergence: The paper provides a rigorous proof that Laplace gating achieves a parametric convergence rate of $O(n^{-1/4})$ for gating and expert parameters, which is significantly faster than the rates typically seen with Softmax gating.
Figure 1: An example of addressing the challenge of FlexiModal Data: patients in ICUs often have extensive and irregular health status measurements over time; patients with milder conditions only require monitoring across fewer categories. FuseMoE is adept at handling inputs featuring any combination of modalities, including those with missing elements. It starts by encoding inputs using modality-specific feature extractors, followed by employing a multi-time attention mechanism [Shukla and Marlin, 2021] to address temporal irregularities. The core of FuseMoE lies the MoE Fusion Layer, where a routing mechanism is trained to categorize multimodal inputs and direct them to the appropriate combinations of MLPs. The outputs from these MLPs are weighted through a gating function, resulting in fused embeddings, which are subsequently utilized for further processing.

Key Results

  • Clinical Superiority: On the MIMIC-IV mortality prediction task, FuseMoE with Laplace gating achieved an AUROC of 81.03, outperforming the state-of-the-art HAIM pipeline (79.65) and standard Softmax-MoE (79.49).
  • Effective Scaling: Unlike baselines that struggled to integrate additional modalities, FuseMoE showed significant performance gains when adding CXR and ECG data to the standard vitals and notes.
  • Robustness to Missingness: In scenarios with missing modalities, the per-modality router configuration effectively mitigated performance drops, even surpassing models trained on complete data in some instances by better separating present vs. absent information.

Limitations and Future Work

The authors note that their current approach to encoding temporal irregularities (mTAND) can lead to over-parameterization when the input sample size is small. They suggest that future research should focus on developing simpler, more efficient methods for handling irregular sampling that maintain the model's overall performance while reducing the parameter footprint.

Summary updated

Abstract

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.
BibTeX
@inproceedings{han2024fusemoe,
  author = {Xing Han and Huy Nguyen and Carl Harris and Nhat Ho and Suchi Saria},
  title = {{FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=jfE7XCE89y},
}
Figure