Abstract
Multi-modal Federated Learning (FL) in real-world edge environments faces significant challenges due to unlabelled data and missing modalities. TACTFL introduces a novel framework that leverages temporal overlap as a supervisory signal for local self-supervised training and employs a similarity-guided aggregation strategy on the server to mitigate model drift. Experimental results demonstrate that TACTFL significantly outperforms existing baselines, particularly in scenarios with extreme label scarcity and sensor failures.
Introduction
Traditional Federated Learning usually relies on two unrealistic assumptions: that client data is fully labeled and consists of a single modality. In the real world—think smartphones, smartwatches, or IoT cameras—data is highly heterogeneous, mostly unlabelled, and multi-modal (e.g., synchronized video, audio, and accelerometer readings). Furthermore, sensors frequently fail or are turned off, leading to missing modalities. TACTFL addresses the challenge of collaboratively training a global model when clients only have unlabelled, incomplete, multi-modal data, and only the central server has a tiny fraction of labeled data.
Methods
TACTFL (TemporAl Contrastive Training for multi-modal Federated Learning) splits the training process into two distinct phases:
Local Temporal Contrastive Training (Client-side)
Instead of relying on labels, TACTFL uses time as the supervisory signal. It splits multi-modal sequences into overlapping temporal windows. Using a metric called temporal Intersection-over-Union (tIoU), it creates soft pseudo-labels. Embeddings from different modalities (e.g., audio and video) that share high temporal overlap are pushed together in the feature space, while non-overlapping ones are pushed apart. This is completely modality-agnostic and requires zero labels.
Similarity-guided Model Aggregation (Server-side)
Because clients train self-supervised on highly varied data, their local models can drift and diverge semantically. Standard aggregation (like FedAvg) often fails in this context. Instead, TACTFL uses the server's small labeled dataset to test each uploaded client encoder. It calculates the cosine similarity between the representations produced by each client. Clients whose feature spaces align well with the majority are given higher weights during aggregation, filtering out noisy or biased local models. Finally, the server freezes these aggregated encoders and trains the downstream task head using its small labeled dataset.
Experimental Setup
The authors evaluate TACTFL across a wide variety of standard benchmarks, adapting them for multi-modal and missing-modality FL scenarios:
- Video/Audio: UCF101, MiT-10
- Wearable/Sensor: UCI-HAR (accelerometer/gyroscope), mHealth, UR-Fall (RGB/depth/sensors), Opportunity
- Image/Text & Audio/Text: Hateful Memes, MELD
Key Results
TACTFL demonstrates massive performance leaps over standard baselines, especially when labeled data is scarce or sensor modalities are missing.
Massive Gains in Low-Label Regimes
On the UCF101 dataset, when only 10% of the data is labeled (and kept on the server), TACTFL achieves 68.48% top-1 accuracy, significantly outperforming the FedOpt baseline of 35.35%. This resilience to label scarcity is consistent across multiple datasets and metrics.
Robustness to Missing Sensors
In scenarios where up to 80% of modalities are randomly dropped across clients (simulating hardware failures), TACTFL still outperforms state-of-the-art methods. For example, on the MELD dataset with an 80% missing modality rate, TACTFL hits 53.25% accuracy compared to the previous best of 52.18% (MFCPL). In extreme ablation studies combining 90% missing labels and 90% missing modalities, TACTFL maintained a remarkable 72.22% accuracy on UCF101, whereas standard FedAvg collapsed to just 10.49% under the missing label condition alone.
Conclusion
TACTFL proves that temporal synchronization is a powerful substitute for human labels in multi-modal alignment. By combining soft-label contrastive learning with representational similarity aggregation, the framework achieves high privacy and low communication overhead without requiring cross-client data sharing. Future work will explore dynamic temporal alignment techniques and personalized model adaptation for highly heterogeneous client populations.
Summary updated