On the Effectiveness of Partial Variance Reduction in Federated Learning with Heterogeneous Data

Bo Li, Mikkel N. Schmidt, Tommy S. Alstrøm, Sebastian U. Stich

Abstract

Federated learning (FL) often struggles when clients possess highly diverse data distributions, a phenomenon known as "client drift" that hampers global model convergence. While existing variance reduction methods like SCAFFOLD mitigate this drift, they typically double the communication overhead, making them impractical for bandwidth-constrained environments. This paper introduces FedPVR (Partial Variance Reduction), a method that applies gradient correction selectively to the final classification layers of deep neural networks. We demonstrate that early feature extraction layers are naturally robust to heterogeneity, and that stabilizing only the "head" of the model achieves up to a 6.7x speedup over standard FedAvg with significantly lower communication costs.

Introduction

Federated Learning (FL) enables collaborative training of machine learning models across decentralized clients without raw data exchange. The standard algorithm, FedAvg, performs well under independent and identically distributed (IID) data. However, in real-world scenarios where data is heterogeneous (non-IID), local updates diverge from the global objective. This "client drift" slows convergence and degrades final accuracy.

Previous attempts to solve this using full variance reduction (VR) require transmitting control variates for every model parameter. For deep networks, this doubling of communication volume is often prohibitive. We propose a more efficient alternative based on the observation that drift is not uniform across model layers.

The Empirical Basis for Partial Correction

To understand where drift occurs, we analyzed deep models using Centered Kernel Alignment (CKA) and a novel "Drift Diversity" metric. Our findings reveal a stark contrast between early and late layers: feature extraction layers maintain high similarity across clients even under extreme heterogeneity, whereas the final classification layers exhibit significant divergence.

Figure 2: Analysis of data distribution (left), CKA similarity (middle), and Drift Diversity (right) across different layers and communication rounds. Higher drift is observed in deeper layers and the classifier.

This insight suggests that correcting the entire model is unnecessary and potentially counterproductive. By allowing early layers to remain "diverse," they can learn richer representations that benefit the global model, provided the final decision-making "head" is stabilized.

FedPVR: Partial Variance Reduction

FedPVR applies variance reduction selectively. It maintains control variates—states that track the difference between local and global update directions—only for a subset of the model's final layers.

Figure 1: (Left) The FedPVR architecture applying variance reduction only to the classifier. (Right) Test accuracy vs. communication rounds for CIFAR-10 and CIFAR-100 under high heterogeneity.

By using a binary mask to restrict correction to the "head," FedPVR reduces the communication overhead from the 4x required by SCAFFOLD to approximately 2.1x relative to FedAvg, while maintaining or exceeding the convergence speed of full-model correction.

Experimental Evaluation

We evaluated FedPVR on CIFAR-10 and CIFAR-100 datasets using VGG-11 and ResNet-8 architectures. Data was partitioned using a Dirichlet distribution ($\alpha$) to simulate varying degrees of heterogeneity.

Key Results

  • Performance Gains: FedPVR achieved a 1.5x to 6.7x speedup over FedAvg. On CIFAR-100 with extreme heterogeneity ($\alpha=0.1$), FedPVR reached 43.5% accuracy, outperforming FedAvg (34.3%) and FedProx (35.0%).
  • Communication Efficiency: FedPVR achieves competitive accuracy with roughly half the extra data transmission required by state-of-the-art variance reduction methods.
  • Regularization Effect: In moderate heterogeneity settings, FedPVR occasionally outperformed centralized training, suggesting that the stabilized distributed updates provide a beneficial regularizing effect.

Conclusion

FedPVR demonstrates that targeted variance reduction is a powerful and efficient strategy for heterogeneous federated learning. By focusing correction on the classification "head," we resolve client drift while preserving the beneficial feature diversity of early layers. Future work includes developing adaptive methods to automatically select the optimal layers for correction across varying architectures and data distributions.

Summary updated

Abstract

Data heterogeneity across clients is a key challenge in federated learning. Prior works address this by either aligning client and server models or using control variates to correct client model drift. Although these methods achieve fast convergence in convex or simple non-convex problems, the performance in over-parameterized models such as deep neural networks is lacking. In this paper, we first revisit the widely used FedAvg algorithm in a deep neural network to understand how data heterogeneity influences the gradient updates across the neural network layers. We observe that while the feature extraction layers are learned efficiently by FedAvg, the substantial diversity of the final classification layers across clients impedes the performance. Motivated by this, we propose to correct model drift by variance reduction only on the final layers. We demonstrate that this significantly outperforms existing benchmarks at a similar or lower communication cost. We furthermore provide proof for the convergence rate of our algorithm.
BibTeX
@inproceedings{li2023effectiveness,
  author = {Bo Li and Mikkel N. Schmidt and Tommy S. Alstrøm and Sebastian U. Stich},
  title = {{On the Effectiveness of Partial Variance Reduction in Federated Learning with Heterogeneous Data}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00386},
}
Figure