Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities

Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin, Sung-Ju LeeSubmission to ICLR 2025 · Withdrawn

TL;DR

Wearable healthcare applications often rely on federated learning (FL) across devices with missing or incomplete sensor streams (e.g., missing heart rate or accelerometer channels). While existing approaches use heavy multi-encoder setups or complex data imputation that scale poorly as sensor counts grow, FLISM adopts an efficient early-fusion strategy enhanced with contrastive learning, entropy-weighted aggregation, and knowledge distillation. This achieves state-of-the-art accuracy while slashing communication overhead by up to $79\times$ and computation by up to $41\times$.

What Problem Does This Paper Address?

Personal mobile and wearable devices collect diverse time-series biosignals (like ECG, EDA, PPG, and accelerometer readings) that are ideal for collaborative, privacy-preserving federated learning. In the real world, however, clients rarely possess the complete set of sensing modalities due to battery constraints, hardware differences, or sensor faults.

Prior FL approaches handle missing modalities using:

  1. Intermediate fusion: Training a separate encoder for each modality before fusing features. This approach scales poorly—communication and compute costs explode as the number of supported sensors increases.
  2. Deep imputation: Training cross-modality generative models (like autoencoders). This requires $M(M-1)$ cross-modal models for $M$ modalities, which quickly becomes computationally intractable.

Early fusion (concatenating all raw sensor streams into a single model) is lightweight and captures cross-sensor interactions early. But in an FL setting where raw data cannot leave client devices, missing modalities must be filled with zeros. Without intervention, simple zero-imputation causes severe feature distribution shifts and client update drift, degrading global model performance.

Methods

FLISM (Federated Learning with Incomplete Sensing Modalities) tackles the pitfalls of zero-imputed early fusion using three complementary components:

  1. Modality-Invariant Representation Learning (MIRL): To prevent the unified encoder from overfitting to specific modality subsets, clients apply supervised contrastive learning (SupCon) locally. Augmented views are created via modality dropout (randomly zeroing out up to $m_k - 1$ modalities) plus Gaussian noise. This forces the encoder to cluster representations of identical labels together regardless of which sensors were active.
  2. Modality Quality-Aware Aggregation (MQAA): Instead of standard sample-weighted aggregation (FedAvg), the central server weights each client’s update by the inverse of its prediction entropy: $$r_k^t = \left( -\frac{1}{n_k} \sum_{i=1}^{n_k} \sum_{j=1}^{|C|} p_{i,j} \log p_{i,j} \right)^{-1}$$ Clients with more complete, informative modalities make higher-confidence (lower entropy) predictions, naturally granting them higher influence in the global model aggregation.

  3. Global-Aligned Knowledge Distillation (GAKD): Clients with limited or lower-quality modalities can easily diverge and produce biased local updates. During local optimization, clients distill knowledge from the previous global model ($W_{t-1}$) using Kullback–Leibler divergence on softened logits, anchoring local updates to the generalized global consensus.

Datasets

The authors evaluated FLISM across four real-world, publicly available multimodal healthcare time-series benchmarks:

  • PAMAP2: 6 modalities (accelerometer and gyroscope at wrist, chest, ankle; 12 physical activities).
  • RealWorld HAR: 10 modalities (accelerometer and gyroscope across 5 body locations; 8 activities).
  • WESAD: 10 modalities (chest and wrist sensors measuring ECG, EDA, EMG, BVP, respiration, temperature, and acceleration for 3-class affect detection).
  • Sleep-EDF: 5 modalities (EEG channels, EOG, chin EMG, and respiration across 5 sleep stages).

Key Insights and Ideas

  • Label skew solutions don't fix modality skew: Popular non-IID FL algorithms like FedProx and MOON struggle or perform worse than standard FedAvg under incomplete modality conditions, showing that modality deficiency induces a fundamentally distinct distribution shift from label imbalance.
  • Prediction entropy is an effective proxy for modality quality: The authors empirically demonstrate that lower prediction entropy on local data strongly correlates with higher modality completeness and downstream test accuracy, providing a zero-overhead metric for aggregation weighting.
  • Early fusion can outperform intermediate fusion without the overhead: By pairing modality dropout contrastive learning with global distillation, a single early-fusion backbone avoids the parameter explosion of unimodal encoders while avoiding catastrophic representation drift.

Key Results

  • Accuracy Gains: Across missing modality ratios ($p \in \{40\%, 60\%, 80\%\}$), FLISM outperforms early-fusion baselines (FedAvg, FedProx, MOON) by 0.043 to 0.143 in macro F1 score. It also surpasses intermediate fusion baselines (FedMM, Harmony) by 0.037 to 0.055 average F1 score.
  • Efficiency over Intermediate Fusion: Compared to FedMM, FLISM reduces communication latency by $2.70\times$–$3.16\times$ and compute costs by $2.81\times$–$3.36\times$. Compared to Harmony, it communicates updates $1.13\times$–$7.01\times$ faster.
  • Efficiency over Deep Imputation: Against an extended multi-modality variant of AutoFed (AutoFed+), FLISM achieves $0.067$–$0.078$ higher F1 score while being $75.85\times$–$79.15\times$ more communication-efficient and requiring $29.31\times$–$41.28\times$ fewer computation operations.
  • Scalability: In synthetic scaling benchmarks simulating up to 30 modalities, FLISM maintains virtually constant communication and computation footprints, whereas intermediate fusion baselines scale steeply (up to 7.0 billion parameters and hours of extra communication).

Limitations and Future Work

  • 1D Sensing Focus: FLISM is tailored to 1D time-series data; extending early fusion to high-dimensional modalities (e.g., paired video, audio, or text) will likely require pre-trained feature extraction backbones.
  • Static vs. Dynamic Modality Drops: The current experimental setup assumes modalities are missing statically throughout training rather than dropping dynamically at arbitrary runtime intervals.
  • Potential Bias in Extreme Scenarios: If high-quality clients are rare or correlate heavily with specific demographic groups, entropy-weighted aggregation (MQAA) could theoretically over-represent those subgroups, requiring explicit fairness constraints.
  • System Heterogeneity: Client selection currently does not incorporate device-level hardware constraints (battery life, network connectivity, or compute budget).

Summary updated

Abstract

Many healthcare sensing applications utilize multimodal time-series data from sensors embedded in mobile and wearable devices. Federated Learning (FL), with its privacy-preserving advantages, is particularly well-suited to such applications. However, most multimodal FL methods assume the availability of complete modality data for local training, which is often unrealistic. Moreover, recent approaches for tackling incomplete modalities scale poorly and become inefficient as the number of modalities increases. To address these limitations, we propose FLISM, an innovative algorithm that enables efficient FL training with incomplete sensing modalities while maintaining high accuracy. FLISM employs three key techniques: (1) modality-invariant representation learning to extract effective features from clients with a diverse set of modalities, (2) modality quality-aware aggregation to prioritize contributions from clients with higher-quality data, and (3) global-aligned knowledge distillation to mitigate local update shifts caused by modality differences. Extensive experiments on real-world datasets show that FLISM not only achieves high accuracy but is also faster and more efficient compared to state-of-the-art methods designed to handle incomplete modality problems in FL.
BibTeX
@misc{orzikulova2024federated,
  author = {Adiba Orzikulova and Jaehyun Kwak and Jaemin Shin and Sung-Ju Lee},
  title = {{Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities}},
  year = {2024},
  eprint = {2405.11828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.11828},
}
Figure