TL;DR
FuseMoE is a novel Mixture-of-Experts (MoE) framework designed to handle "FlexiModal" data—multimodal inputs characterized by irregular temporal sampling and frequent missing values. By introducing a new Laplace gating function, the authors provide both a theoretical guarantee of faster convergence and empirical proof of superior performance across clinical, sentiment, and activity recognition tasks.
What Problem Does This Paper Address?
In high-stakes fields like healthcare, machine learning models rarely encounter "clean" data. Instead, they face FlexiModal Data: a messy mix of modalities (e.g., clinical notes, vital signs, X-rays) where samples are collected at irregular intervals and many modalities are simply missing for certain individuals.
Existing multimodal fusion techniques often fall short because they:
- Lack Scalability: Many are designed for only two modalities or rely on computationally expensive pairwise comparisons.
- Struggle with Missingness: They either require all modalities to be present or rely on suboptimal data imputation.
- Ignore Irregularity: They often fail to account for the varying temporal dynamics inherent in real-world time-series data.
Methods
FuseMoE tackles these challenges by combining a specialized encoder with a sparse MoE backbone.
- Modality and Irregularity Encoder: The model uses a discretized multi-time attention (mTAND) module to map irregularly sampled observations into a standardized, regularly sampled embedding space.
- Sparse MoE Backbone: Instead of a single dense layer, FuseMoE uses a set of Feed-Forward Network (FFN) "experts." A router selects the best experts for a given input, allowing the model to scale capacity without a massive increase in computational cost.
- The Laplace Gating Function: This is the paper's primary innovation. Traditional MoE models use a Softmax gating function based on the inner product of inputs and expert weights. FuseMoE uses a Laplace gating function based on $L_2$ (Euclidean) distance: $h_l(x) = \text{Top K}(-\|W - x\|_2)$.
- Flexible Routing Strategies: The authors explore three router designs to optimize how different modalities interact with experts:
- Joint Router: All modalities are concatenated and sent to a shared pool of experts.
- Per-modality Routers: Each modality has its own router but shares a common pool of experts.
- Disjoint Experts & Routers: Each modality has its own router and its own dedicated pool of specialized experts.
Datasets
The authors evaluated FuseMoE on a diverse range of benchmarks to prove its versatility:
- Clinical (MIMIC-III & MIMIC-IV): Electronic Health Records containing vitals, notes, X-rays, and ECGs. Tasks included mortality prediction (48-IHM), length-of-stay (LOS), and phenotype classification (25-PHE).
- Sentiment Analysis (CMU-MOSI & MOSEI): Multimodal data consisting of video, audio, and text.
- Activity Recognition (PAM): Data from 17 sensors monitoring daily living activities.
- Image Classification (CIFAR-10): Used specifically to validate the Laplace gating function within a Vision-MoE framework.
Key Insights and Ideas
- Balanced Expert Selection: Softmax gating can lead to "representation collapse," where a few experts dominate because the inner product is sensitive to magnitude. Laplace gating, by using Euclidean distance, ensures a more balanced selection process, which is crucial when dealing with heterogeneous multimodal features.
- Handling Missing Data Dynamically: By using per-modality routers and a learnable "missing indicator" embedding, FuseMoE can dynamically down-weight the influence of experts responsible for absent data without needing explicit imputation.
- Theoretical Convergence: The paper provides a rigorous proof that Laplace gating achieves a parametric convergence rate of $O(n^{-1/4})$ for gating and expert parameters, which is significantly faster than the rates typically seen with Softmax gating.
Key Results
- Clinical Superiority: On the MIMIC-IV mortality prediction task, FuseMoE with Laplace gating achieved an AUROC of 81.03, outperforming the state-of-the-art HAIM pipeline (79.65) and standard Softmax-MoE (79.49).
- Effective Scaling: Unlike baselines that struggled to integrate additional modalities, FuseMoE showed significant performance gains when adding CXR and ECG data to the standard vitals and notes.
- Robustness to Missingness: In scenarios with missing modalities, the per-modality router configuration effectively mitigated performance drops, even surpassing models trained on complete data in some instances by better separating present vs. absent information.
Limitations and Future Work
The authors note that their current approach to encoding temporal irregularities (mTAND) can lead to over-parameterization when the input sample size is small. They suggest that future research should focus on developing simpler, more efficient methods for handling irregular sampling that maintain the model's overall performance while reducing the parameter footprint.
Summary updated