Workie-Talkie: Accelerating Federated Learning by Overlapping Computing and Communications via Contrastive Regularization

Rui Chen, Qiyu Wan, Pavana Prakash, Lan Zhang, Xu Yuan, Yanmin Gong, Xin Fu, Miao Pan

Abstract

Standard Federated Learning (FL) often suffers from significant idle times due to the "stop-and-wait" nature of synchronous updates, particularly in communication-constrained mobile edge environments. We present FedCR (Federated Contrastive Regularization), nicknamed "Workie-Talkie," a framework designed to overlap local computation with wireless communication. FedCR addresses the resulting challenges of model staleness and data heterogeneity through a novel class-wise contrastive regularization mechanism. Our approach supports heterogeneous hardware by enabling subnetwork training with fixed projection heads. Experimental results on CIFAR and Tiny-ImageNet benchmarks demonstrate that FedCR achieves up to a 2.01x speedup in training latency and significant accuracy improvements over state-of-the-art asynchronous FL methods.

Introduction: The Communication Bottleneck in FL

The widely adopted Federated Averaging (FedAvg) algorithm follows a synchronous execution pattern: edge devices perform local training, transmit updates to a central server, and remain idle until the aggregated global model is returned. In real-world mobile edge deployments, wireless communication often becomes a severe bottleneck, leading to low resource utilization and slow convergence.

While overlapping computation and communication can mitigate these delays, it introduces two critical issues:

  1. Model Staleness: Devices continue training on local weights while the server processes the previous round's updates, leading to gradients derived from "stale" global information.
  2. Data Heterogeneity (Non-IID): Local models tend to drift toward local optima, a problem exacerbated when updates are asynchronous and based on heterogeneous local datasets.
Visualizing client model similarity and drift under different staleness levels ($S=0$ vs $S=40$).

Methodology: The FedCR Framework

FedCR transforms the FL process into a "Workie-Talkie" system where communication and computation occur concurrently. The core of the framework is a specialized contrastive regularization objective that anchors local updates to the global model.

The FedCR architecture featuring feature extraction, fixed projectors, contrastive regularization, and adaptive temperature assignment.

Contrastive Regularization (CR)

To counteract the drift caused by staleness, FedCR introduces a contrastive loss term. This loss pulls the features generated by the local model closer to those of the global model while pushing them away from outdated local features. This "spatial anchoring" ensures that even as devices work ahead, they remain aligned with the global objective.

Class-wise Temperature Scaling

Recognizing that data heterogeneity affects classes differently, FedCR employs an adaptive regularization strategy. The server uses a small auxiliary dataset to evaluate the global model's performance per class. It then distributes class-specific "temperature" parameters to devices, which adjust the strength of the contrastive correction for each class during local training.

Support for Hardware Heterogeneity

FedCR accommodates a diverse fleet of devices (e.g., NVIDIA Jetson Nano to Xavier NX) by allowing them to train "subnetworks"—smaller architectural slices of the full global model. To maintain feature compatibility across different subnetwork sizes, FedCR utilizes a fixed, non-trainable projection head. This ensures that features from various model scales can be aligned within the same contrastive embedding space without additional training overhead.

Experimental Evaluation

The framework was evaluated using CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. To simulate realistic edge conditions, data was partitioned using a Dirichlet distribution to create high non-IID variance.

Key Results

  • Training Speedup: FedCR reached target accuracy levels up to 2.01x faster than LGC and 1.52x faster than the DGA approach on CIFAR-100.
  • Accuracy Improvements: Under high-latency conditions, FedCR improved final test accuracy by 5.8% compared to DGA.
  • Resource Efficiency: Unlike previous methods that require storing multiple model versions to handle staleness, FedCR maintains a constant and low memory footprint, making it ideal for resource-constrained edge devices.
  • Subnetwork Performance: In heterogeneous hardware setups, FedCR boosted the test accuracy of smaller subnetworks by 23% to 37% compared to existing heterogeneous FL baselines.

Conclusion

FedCR effectively "hides" communication latency by overlapping it with local computation. By leveraging contrastive learning as a regularization tool rather than just a pre-training step, it successfully manages the trade-offs between staleness and convergence. The use of fixed projectors and class-wise adaptive scaling further ensures robustness across heterogeneous data and hardware, paving the way for more efficient large-scale federated learning in the wild.

Summary updated

Abstract

Federated learning (FL) over mobile edge devices is a promising distributed learning paradigm for various mobile applications. However, practical deployment of FL over mobile devices is very challenging because (i) conventional FL incurs huge training latency for mobile edge devices due to interleaved local computing and communications of model updates, (ii) there are heterogeneous training data across mobile edge devices, and (iii) mobile edge devices have hardware heterogeneity in terms of computing and communication capabilities.To address aforementioned challenges, in this paper, we propose a novel "workie-talkie" FL scheme, which can accelerate FL’s training by overlapping local computing and wireless communications via contrastive regularization (FedCR). FedCR can reduce FL’s training latency and almost eliminate straggler issues since it buries/embeds the time consumption of communications into that of local training. To resolve the issue of model staleness and data heterogeneity co-existing, we introduce class-wise contrastive regularization to correct the local training in FedCR. Besides, we jointly exploit contrastive regularization and subnetworks to further extend our FedCR approach to accommodate edge devices with hardware heterogeneity. We deploy FedCR in our FL testbed and conduct extensive experiments. The results show that FedCR outperforms its status quo FL approaches on various datasets and models.
BibTeX
@inproceedings{chen2023workie,
  author = {Rui Chen and Qiyu Wan and Pavana Prakash and Lan Zhang and Xu Yuan and Yanmin Gong and Xin Fu and Miao Pan},
  title = {{Workie-Talkie: Accelerating Federated Learning by Overlapping Computing and Communications via Contrastive Regularization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01559},
}
Figure