Introduction
Federated Learning (FL) has emerged as a privacy-preserving paradigm for training models across distributed IoT devices. However, the inherent system heterogeneity of these devices—varying in computational power and network bandwidth—introduces the "straggler" problem. In synchronous FL, the server must wait for the slowest device to complete its local update, leading to significant idle time and training inefficiency. While this is a known challenge in unimodal FL, it becomes particularly severe in Multimodal Federated Learning (MFL). Tasks like video-text retrieval involve complex cross-modal interactions and heavy data structures, making the performance degradation and time overhead caused by stragglers much more significant than in single-modality tasks.
The MFL-AKD Framework
To address these challenges, the authors propose Multimodal Federated Learning with Accelerated Knowledge Distillation (MFL-AKD). This framework mitigates the impact of stragglers by leveraging a powerful centralized teacher model to "warm up" local models through a two-stage training process.
The core components of MFL-AKD include:
- Fast Knowledge Transfer Mechanism (FKTM): To prevent slow devices from delaying the entire round, each client performs an "early training" phase on a small minibatch of data, scaled to its specific hardware capacity. This ensures synchronization is maintained without wasting time on slow updates.
- Centralized Knowledge Distillation (KD): These early-trained models are sent to the server, which houses a frozen, large-scale pretrained vision-language model (CLIP). The server acts as a teacher, distilling high-level multimodal representations into the client "student" models.
- Local Regular Training: The distilled models, now enriched with global multimodal knowledge, are returned to the clients. This "prior knowledge" acts as a shortcut, allowing the models to converge much faster during the subsequent full-batch local training phase.
Experimental Evaluation
The framework was evaluated on four major multimodal benchmarks: Flickr30k and MSCOCO for image-text retrieval, and Charades-STA and ActivityNet Captions for video moment retrieval. These datasets represent the high-dimensional complexity typical of MFL tasks that are sensitive to straggler effects.
Key Results and Insights
MFL-AKD consistently outperforms state-of-the-art baselines such as FedVMR and FedGEMS across all datasets. A critical finding is the framework's robustness to stragglers. As simulated "sleep times" (delays) increase, MFL-AKD maintains significantly lower training times compared to standard methods.
On the Flickr30k dataset, MFL-AKD completed training in approximately 20.3k seconds under high-delay conditions, whereas standard FedAvg required 31.3k seconds—a massive efficiency gain. Furthermore, the distillation process enables faster convergence, reaching higher accuracy in fewer communication rounds. This efficiency stems from the "warm-up" advantage, where local models start their main training phase with a superior initialization derived from CLIP's global knowledge.
Conclusion and Future Work
MFL-AKD effectively tackles system heterogeneity in MFL by combining workload balancing with server-side knowledge distillation. By utilizing the server's superior compute power to boost local models, the framework ensures that even the slowest devices contribute effectively without bottlenecking the system. Future research may explore the framework's sensitivity to domain shifts between the teacher model and local data, as well as its performance across heterogeneous model architectures.
Summary updated