Abstract
Many mobile sensing applications utilize data from various modalities, including motion and physiological sensors in mobile and wearable devices. Federated Learning (FL) is particularly suitable for these applications thanks to its privacy-preserving feature. However, challenges such as limited battery life, poor network conditions, and sensor malfunctions can restrict the use of all available modalities for local model training. Additionally, existing multimodal FL systems also struggle with scalability and efficiency as the number of modality sources increases. To address these issues, we introduce FLISM, a framework designed to enable multimodal FL with incomplete modalities. FLISM leverages simulation technique to learn robust representations that can handle missing modalities and transfers model knowledge across clients with varying set of modalities. The evaluation results using three real-world datasets and simulations demonstrate FLISM's effective balance between model performance and system efficiency. It shows an average improvement of .067 in F1-score, while also reducing communication (2.69x faster) and computational (2.28x more efficient) overheads compared to existing methods addressing incomplete modalities. Moreover, in simulated scenarios involving tasks with a larger number of modalities, FLISM achieves a significant speedup of 3.23x~85.10x in communication and 3.73x~32.29x in computational efficiency.
Introduction
Multimodal healthcare sensing—utilizing data from accelerometers, heart rate monitors, and skin temperature sensors—is a powerful tool for longitudinal health tracking. Federated Learning (FL) is the natural choice for these applications, as it enables model training on personal devices without exposing sensitive health data. However, a significant practical hurdle remains: incomplete modalities.
In real-world deployments, sensors often fail, users forget to wear specific devices, or systems disable high-power sensors to conserve battery. Most existing FL algorithms assume a complete set of data across all clients. Current solutions typically rely on "intermediate fusion" (separate encoders for each sensor) or "deep imputation" (generating synthetic data to fill gaps), both of which are computationally expensive and scale poorly as the number of sensors increases.
The FLISM Framework
To overcome these limitations, we propose FLISM (Federated Learning with Incomplete Sensing Modalities). FLISM prioritizes early fusion—combining sensor data at the input level—to maintain maximum efficiency on resource-constrained mobile devices. To make early fusion robust against missing data, the framework integrates three core components:
- Modality-Invariant Representation Learning (MIRL): By employing supervised contrastive learning (SupCon) and "modality dropout," FLISM teaches the model to extract consistent features regardless of which sensors are active. This ensures that the underlying activity (e.g., "walking") is recognized whether the input is heart rate, movement, or both.
- Modality Quality-Aware Aggregation (MQAA): Recognizing that not all client updates are equally informative, FLISM uses prediction entropy as a proxy for data quality. Clients with more complete data produce more confident (low-entropy) predictions; the server assigns these updates higher weights during global aggregation, preserving privacy without needing to know a client's specific sensor configuration.
- Global-Aligned Knowledge Distillation (GAKD): To mitigate local model drift caused by specific missing sensors, FLISM uses the global model as a teacher. Local models are penalized if their predictions deviate significantly from the generalized global model, ensuring alignment across the network.
Experimental Evaluation
We evaluated FLISM using four diverse, real-world multimodal time-series datasets: PAMAP2 (physical activity), WESAD (stress detection), RealWorld HAR (activity recognition), and Sleep-EDF (sleep stage classification). These datasets range from 5 to 10 distinct modalities.
Performance and Efficiency
FLISM consistently outperformed both standard FL baselines and specialized multimodal methods like FedMM and Harmony. Key results include:
- Accuracy Gains: FLISM achieved F1 score improvements of .043 to .143 over early fusion baselines and .037 to .055 over state-of-the-art intermediate fusion methods.
- System Efficiency: Compared to intermediate fusion, FLISM was 3.11× faster in communication and 2.14× more efficient in terms of trained parameters.
- Scalability: Against deep imputation methods (e.g., AutoFed+), FLISM demonstrated a 77× reduction in communication overhead and a 35× reduction in computational costs.
Conclusion and Future Work
FLISM demonstrates that early fusion, when combined with representation learning and quality-aware aggregation, is the superior choice for multimodal FL on mobile devices. While the current framework is optimized for 1D time-series data and static missing modalities, future work will explore dynamic sensor drops and extending these techniques to high-dimensional data like video and audio. As wearable ecosystems expand to include dozens of sensors, the linear scalability of FLISM will be essential for practical healthcare applications.
Summary updated