TL;DR
FedMFS introduces a selective modality communication strategy for multimodal federated learning that enables resource-constrained edge devices to intelligently choose which sensor models to upload to a central server. By balancing a modality's performance impact (via Shapley values) against its communication cost (model size), FedMFS achieves up to 97.34% accuracy on real-world datasets while slashing communication overhead by over 4x compared to traditional baselines.
What Problem Does This Paper Address?
Federated Learning (FL) allows distributed edge devices—like smartphones, autonomous vehicles, and IoT sensors—to collaboratively train machine learning models without sharing their raw, private data. As edge devices become increasingly equipped with diverse sensors (e.g., cameras, LiDAR, microphones, and IMUs), Multimodal Fusion Federated Learning (MFFL) has emerged to leverage these rich, multi-sensory data streams.
However, deploying MFFL in the real world faces two major bottlenecks:
- Modality Heterogeneity: Not every device possesses the same set of sensors due to hardware variations, malfunctions, or environmental factors.
- Communication Constraints: Uploading multiple deep learning models (one for each data modality) from a power- and bandwidth-constrained edge device to a central server creates massive network congestion.
Existing MFFL frameworks typically require parallel processing and full transmission of all available modality models, which is highly inefficient. The authors of this paper ask a fundamental question: How can a client evaluate and select the best subset of modalities to upload, optimizing the trade-off between model accuracy and communication cost?
Methods
The authors propose FedMFS (Federated Multimodal Fusion learning with Selective modality communication). Instead of fusing data at the raw input or feature level, FedMFS utilizes a decision-level fusion approach.
Here is how the workflow operates locally on each client:
- Base Modality Models: Each client trains independent, modality-specific models (e.g., LSTMs for sequential sensor data) on their available local data.
- Personalized Ensemble Model: The definitive predictions from these base models are fed into a local post-processing ensemble model (a Random Forest classifier) to generate the final prediction.
- Selective Modality Communication: To minimize upload overhead, clients compute a composite Priority Score ($P^k_m$) for each modality based on two normalized metrics:
- Shapley Value ($\phi$): A game-theoretic metric used to quantify the exact marginal contribution (impact) of each modality model on the final ensemble prediction.
- Model Size ($|\text{\theta}|$): A proxy for the communication overhead required to transmit that specific model.
- Top-$\gamma$ Selection: The client only uploads the top-$\gamma$ highest-priority modality models to the server for global aggregation, keeping the ensemble model and non-selected models private.
Datasets
The framework was evaluated using ActionSense, a comprehensive, real-world multimodal dataset that tracks human daily kitchen activities using various wearable and environmental sensors.
The experiments utilized six distinct sensor modalities:
- Eye Tracking (Position)
- Myo EMG (Left and Right Arm muscle activity)
- Tactile Gloves (Left and Right Hand pressure data)
- Xsens (Full-body motion tracking/rotation)
Key Insights and Ideas
- Shapley Values as a Pruning Tool: While Shapley values are traditionally used for Explainable AI (XAI) to interpret model decisions post-hoc, FedMFS cleverly repurposes them during the training loop as an active filtering mechanism for communication efficiency.
- More Modalities $\neq$ Better Performance: Aggregating every single modality model isn't always necessary. In fact, sending noisy or high-overhead models can degrade learning efficiency under tight communication constraints.
- Dynamic Modality Importance: The value of a sensor changes over time. Early in the FL process, lightweight modalities (like Eye Tracking) are favored due to low communication costs. As training progresses, more complex but highly informative modalities (like Xsens and Right-Arm Myo) naturally take over as primary contributors to accuracy.
Key Results
FedMFS was benchmarked against four traditional MFFL baselines (Data-level, Feature-level, Decision-level fusion, and the random-selection FLASH framework).
- Superior Accuracy & Efficiency: Under a strict cumulative communication budget of 50 MB, FedMFS achieved a peak accuracy of 97.34% while requiring only 0.72 MB of communication overhead per iteration (configured with $\gamma=1, \alpha_s=0.2, \alpha_c=0.8$).
- Massive Bandwidth Reduction: Compared to standard decision-level fusion, FedMFS achieved significantly higher accuracy while reducing the communication overhead by nearly an order of magnitude, demonstrating much faster convergence.
Limitations and Future Work
While FedMFS marks a significant step forward for efficient multimodal FL, a few areas remain open for improvement:
- Static Hyperparameters: The weights balancing performance vs. communication ($\alpha_s$ and $\alpha_c$) are currently predetermined and static. The authors note that future iterations should dynamically adjust these weights based on real-time network bandwidth fluctuations.
- Shapley Complexity: Computing exact Shapley values is combinatorially expensive. Although the authors mitigated this via data subsampling (50 samples), scaling this to massive datasets or highly complex ensemble architectures could present local computational challenges.
- Permanent Modality Pruning: The authors suggest that future work could completely discard consistently underperforming modalities (such as the Left-Arm Myo sensor in this specific kitchen task) to optimize local device computation alongside communication.
Summary updated