Abstract
Standard Federated Learning (FL) often suffers from significant idle times due to the "stop-and-wait" nature of synchronous updates, particularly in communication-constrained mobile edge environments. We present FedCR (Federated Contrastive Regularization), nicknamed "Workie-Talkie," a framework designed to overlap local computation with wireless communication. FedCR addresses the resulting challenges of model staleness and data heterogeneity through a novel class-wise contrastive regularization mechanism. Our approach supports heterogeneous hardware by enabling subnetwork training with fixed projection heads. Experimental results on CIFAR and Tiny-ImageNet benchmarks demonstrate that FedCR achieves up to a 2.01x speedup in training latency and significant accuracy improvements over state-of-the-art asynchronous FL methods.
Introduction: The Communication Bottleneck in FL
The widely adopted Federated Averaging (FedAvg) algorithm follows a synchronous execution pattern: edge devices perform local training, transmit updates to a central server, and remain idle until the aggregated global model is returned. In real-world mobile edge deployments, wireless communication often becomes a severe bottleneck, leading to low resource utilization and slow convergence.
While overlapping computation and communication can mitigate these delays, it introduces two critical issues:
- Model Staleness: Devices continue training on local weights while the server processes the previous round's updates, leading to gradients derived from "stale" global information.
- Data Heterogeneity (Non-IID): Local models tend to drift toward local optima, a problem exacerbated when updates are asynchronous and based on heterogeneous local datasets.
Methodology: The FedCR Framework
FedCR transforms the FL process into a "Workie-Talkie" system where communication and computation occur concurrently. The core of the framework is a specialized contrastive regularization objective that anchors local updates to the global model.
Contrastive Regularization (CR)
To counteract the drift caused by staleness, FedCR introduces a contrastive loss term. This loss pulls the features generated by the local model closer to those of the global model while pushing them away from outdated local features. This "spatial anchoring" ensures that even as devices work ahead, they remain aligned with the global objective.
Class-wise Temperature Scaling
Recognizing that data heterogeneity affects classes differently, FedCR employs an adaptive regularization strategy. The server uses a small auxiliary dataset to evaluate the global model's performance per class. It then distributes class-specific "temperature" parameters to devices, which adjust the strength of the contrastive correction for each class during local training.
Support for Hardware Heterogeneity
FedCR accommodates a diverse fleet of devices (e.g., NVIDIA Jetson Nano to Xavier NX) by allowing them to train "subnetworks"—smaller architectural slices of the full global model. To maintain feature compatibility across different subnetwork sizes, FedCR utilizes a fixed, non-trainable projection head. This ensures that features from various model scales can be aligned within the same contrastive embedding space without additional training overhead.
Experimental Evaluation
The framework was evaluated using CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. To simulate realistic edge conditions, data was partitioned using a Dirichlet distribution to create high non-IID variance.
Key Results
- Training Speedup: FedCR reached target accuracy levels up to 2.01x faster than LGC and 1.52x faster than the DGA approach on CIFAR-100.
- Accuracy Improvements: Under high-latency conditions, FedCR improved final test accuracy by 5.8% compared to DGA.
- Resource Efficiency: Unlike previous methods that require storing multiple model versions to handle staleness, FedCR maintains a constant and low memory footprint, making it ideal for resource-constrained edge devices.
- Subnetwork Performance: In heterogeneous hardware setups, FedCR boosted the test accuracy of smaller subnetworks by 23% to 37% compared to existing heterogeneous FL baselines.
Conclusion
FedCR effectively "hides" communication latency by overlapping it with local computation. By leveraging contrastive learning as a regularization tool rather than just a pre-training step, it successfully manages the trade-offs between staleness and convergence. The use of fixed projectors and class-wise adaptive scaling further ensures robustness across heterogeneous data and hardware, paving the way for more efficient large-scale federated learning in the wild.
Summary updated