FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

Liangqi Yuan, Dong-Jun Han, Vishnu Pandi Chellapandi, Stanislaw H. Żak, Christopher G. Brinton

TL;DR

FedMFS introduces a selective modality communication strategy for multimodal federated learning that enables resource-constrained edge devices to intelligently choose which sensor models to upload to a central server. By balancing a modality's performance impact (via Shapley values) against its communication cost (model size), FedMFS achieves up to 97.34% accuracy on real-world datasets while slashing communication overhead by over 4x compared to traditional baselines.

What Problem Does This Paper Address?

Federated Learning (FL) allows distributed edge devices—like smartphones, autonomous vehicles, and IoT sensors—to collaboratively train machine learning models without sharing their raw, private data. As edge devices become increasingly equipped with diverse sensors (e.g., cameras, LiDAR, microphones, and IMUs), Multimodal Fusion Federated Learning (MFFL) has emerged to leverage these rich, multi-sensory data streams.

However, deploying MFFL in the real world faces two major bottlenecks:

  1. Modality Heterogeneity: Not every device possesses the same set of sensors due to hardware variations, malfunctions, or environmental factors.
  2. Communication Constraints: Uploading multiple deep learning models (one for each data modality) from a power- and bandwidth-constrained edge device to a central server creates massive network congestion.

Existing MFFL frameworks typically require parallel processing and full transmission of all available modality models, which is highly inefficient. The authors of this paper ask a fundamental question: How can a client evaluate and select the best subset of modalities to upload, optimizing the trade-off between model accuracy and communication cost?

Methods

The authors propose FedMFS (Federated Multimodal Fusion learning with Selective modality communication). Instead of fusing data at the raw input or feature level, FedMFS utilizes a decision-level fusion approach.

Fig. 1. Schematic representation of traditional multimodal fusion federated learning vs. the proposed FedMFS.

Here is how the workflow operates locally on each client:

  • Base Modality Models: Each client trains independent, modality-specific models (e.g., LSTMs for sequential sensor data) on their available local data.
  • Personalized Ensemble Model: The definitive predictions from these base models are fed into a local post-processing ensemble model (a Random Forest classifier) to generate the final prediction.
  • Selective Modality Communication: To minimize upload overhead, clients compute a composite Priority Score ($P^k_m$) for each modality based on two normalized metrics:
  • Shapley Value ($\phi$): A game-theoretic metric used to quantify the exact marginal contribution (impact) of each modality model on the final ensemble prediction.
  • Model Size ($|\text{\theta}|$): A proxy for the communication overhead required to transmit that specific model.
  • Top-$\gamma$ Selection: The client only uploads the top-$\gamma$ highest-priority modality models to the server for global aggregation, keeping the ensemble model and non-selected models private.

Datasets

The framework was evaluated using ActionSense, a comprehensive, real-world multimodal dataset that tracks human daily kitchen activities using various wearable and environmental sensors.

The experiments utilized six distinct sensor modalities:

  • Eye Tracking (Position)
  • Myo EMG (Left and Right Arm muscle activity)
  • Tactile Gloves (Left and Right Hand pressure data)
  • Xsens (Full-body motion tracking/rotation)
Description of ActionSense Dataset

Key Insights and Ideas

  • Shapley Values as a Pruning Tool: While Shapley values are traditionally used for Explainable AI (XAI) to interpret model decisions post-hoc, FedMFS cleverly repurposes them during the training loop as an active filtering mechanism for communication efficiency.
  • More Modalities $\neq$ Better Performance: Aggregating every single modality model isn't always necessary. In fact, sending noisy or high-overhead models can degrade learning efficiency under tight communication constraints.
  • Dynamic Modality Importance: The value of a sensor changes over time. Early in the FL process, lightweight modalities (like Eye Tracking) are favored due to low communication costs. As training progresses, more complex but highly informative modalities (like Xsens and Right-Arm Myo) naturally take over as primary contributors to accuracy.
Fig. 3. The impact of modality models on the ensemble model’s final pre- diction throughout the FedMFS iteration, exemplified with the configuration γ = 1, αs = 0.2, αc = 0.8.

Key Results

FedMFS was benchmarked against four traditional MFFL baselines (Data-level, Feature-level, Decision-level fusion, and the random-selection FLASH framework).

  • Superior Accuracy & Efficiency: Under a strict cumulative communication budget of 50 MB, FedMFS achieved a peak accuracy of 97.34% while requiring only 0.72 MB of communication overhead per iteration (configured with $\gamma=1, \alpha_s=0.2, \alpha_c=0.8$).
  • Massive Bandwidth Reduction: Compared to standard decision-level fusion, FedMFS achieved significantly higher accuracy while reducing the communication overhead by nearly an order of magnitude, demonstrating much faster convergence.
Accuracy vs Communication Overhead

Limitations and Future Work

While FedMFS marks a significant step forward for efficient multimodal FL, a few areas remain open for improvement:

  • Static Hyperparameters: The weights balancing performance vs. communication ($\alpha_s$ and $\alpha_c$) are currently predetermined and static. The authors note that future iterations should dynamically adjust these weights based on real-time network bandwidth fluctuations.
  • Shapley Complexity: Computing exact Shapley values is combinatorially expensive. Although the authors mitigated this via data subsampling (50 samples), scaling this to massive datasets or highly complex ensemble architectures could present local computational challenges.
  • Permanent Modality Pruning: The authors suggest that future work could completely discard consistently underperforming modalities (such as the Left-Arm Myo sensor in this specific kitchen task) to optimize local device computation alongside communication.

Summary updated

Abstract

Multimodal federated learning (FL) aims to enrich model training in FL settings where devices are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each device will be diverse, and (ii) communication limitations prevent devices from uploading all their locally trained modality models to the server. In this paper, we propose Federated Multimodal Fusion learning with Selective modality communication (FedMFS), a new multimodal fusion FL methodology that can tackle the above mentioned challenges. The key idea is the introduction of a modality selection criterion for each device, which weighs (i) the impact of the modality, gauged by Shapley value analysis, against (ii) the modality model size as a gauge for communication overhead. This enables FedMFS to flexibly balance performance against communication costs, depending on resource constraints and application requirements. Experiments on the real-world ActionSense dataset demonstrate the ability of FedMFS to achieve comparable accuracy to several baselines while reducing the communication overhead by over 4x.
BibTeX
@inproceedings{yuan2024fedmfs,
  author = {Liangqi Yuan and Dong-Jun Han and Vishnu Pandi Chellapandi and Stanislaw H. Żak and Christopher G. Brinton},
  title = {{FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication}},
  booktitle = {ICC},
  year = {2024},
  url = {https://openreview.net/forum?id=9nrurKBBBr},
}
Figure