Abstract
Federated Learning (FL) enables collaborative model training across distributed clients while maintaining data privacy. However, its practical deployment is often hindered by system heterogeneity (varying network bandwidths) and data heterogeneity (non-IID data distributions). This paper presents a dual-strategy framework designed to optimize communication efficiency and model accuracy simultaneously. By implementing Bandwidth-aware Compression Ratio Scheduling (BCRS) and Overlap-aware Parameter Weighted Average (OPWA), the proposed method achieves significant improvements in training speed and final model performance compared to traditional Top-K sparsification techniques.
1. Introduction
In Federated Learning, the global training process is frequently bottlenecked by the communication overhead of transmitting high-dimensional model updates. Two primary challenges emerge in real-world settings: the "Straggler Problem," where slow network connections from a subset of clients delay the entire training round, and the "Data Heterogeneity Problem," where non-IID (Independent and Identically Distributed) data causes compressed updates to lose critical, unique information during server-side aggregation.
Existing compression methods typically apply a uniform sparsity target across all clients. While this reduces the data volume, it fails to account for the varying capabilities of client hardware and the diverse nature of their local datasets. This paper introduces a framework that treats bandwidth as a resource to be optimized and leverages the sparsity patterns of updates to preserve rare, client-specific knowledge.
2. Methodology
The proposed framework integrates two novel algorithms to address the dual challenges of FL:
2.1 Bandwidth-aware Compression Ratio Scheduling (BCRS)
Rather than enforcing a static compression ratio, BCRS dynamically models the uplink time for each client. By identifying the slowest client as the temporal benchmark for a training round, BCRS allows clients with higher bandwidth to transmit more data (i.e., lower compression) within the same timeframe. This approach ensures that no bandwidth is wasted and that the server receives the highest possible fidelity of updates from the available fleet without increasing the total round duration.
2.2 Overlap-aware Parameter Weighted Average (OPWA)
Standard aggregation methods, such as FedAvg, treat all parameter updates equally. However, after sparsification, certain parameters may be updated by many clients (high overlap), while others are updated by only a few (low overlap). The authors observed that low-overlap updates often represent "specialist" knowledge derived from unique local data distributions. OPWA utilizes a parameter mask to identify these rare updates and applies an "enlarge rate" ((\gamma)) to amplify their weight during aggregation. This prevents unique insights from being "washed out" by the majority, effectively mitigating the negative impacts of non-IID data.
3. Experimental Setup
The framework was evaluated using the ResNet18 architecture across three image recognition benchmarks: CIFAR-10, CIFAR-100, and SVHN. To simulate realistic non-IID conditions, the authors employed a Dirichlet distribution to partition the data, creating scenarios with significant label imbalance across clients. The experiments compared the proposed framework against standard FedAvg and fixed-ratio Top-K sparsification.
4. Key Results and Discussion
4.1 Accuracy and Convergence
The combination of BCRS and OPWA demonstrated superior performance in highly heterogeneous environments, outperforming uncompressed FedAvg by up to 13% in top-1 accuracy. Furthermore, OPWA was found to stabilize the training process, reducing the "client shift" often associated with aggressive compression and leading to smoother convergence peaks.
4.2 Communication Efficiency
The BCRS strategy significantly accelerated training. To reach a baseline target accuracy of 40% on the CIFAR-10 dataset, the proposed method was 3.37× faster than standard Top-K compression. This speedup is attributed to the efficient utilization of available bandwidth, allowing the model to ingest more information per unit of time.
4.3 Heuristics for Implementation
The study suggests that the optimal enlarge rate ((\gamma)) for rare parameters is generally proportional to the number of participating clients. This provides a practical heuristic for implementing the OPWA algorithm in production-scale FL systems.
5. Conclusion and Future Work
This research demonstrates that bandwidth-aware scheduling and weighted aggregation can overcome the traditional trade-offs between compression and accuracy in Federated Learning. While the current work focuses on the uplink bottleneck, future research could extend these strategies to downlink compression. Additionally, exploring the application of BCRS and OPWA within Personalized Federated Learning (PFL) or architectures involving heterogeneous model structures remains a promising avenue for further development.
Summary updated