FedProf: Optimizing Federated Learning with Dynamic Data Profiling

Wentai Wu, Ligang He, Weiwei Lin, Rui Mao, Chenlin Huang, Wei Song

Introduction

Federated Learning (FL) has emerged as a critical paradigm for training machine learning models on decentralized data while preserving user privacy. However, a significant bottleneck in practical FL deployments is the presence of heterogeneous data quality across participating clients. Standard algorithms, such as FedAvg, typically treat all client updates with equal weight. In reality, many clients may possess "low-value" data—characterized by sensor noise, blurring, or extreme class bias—which can significantly impede convergence and degrade the final model's performance.

Impact of biased and noisy local data on the accuracy of the global model over communication rounds.

As illustrated in Figure 1, the presence of biased or noisy local data leads to a substantial gap in accuracy and slower convergence compared to ideal, noiseless conditions. Identifying these detrimental clients without inspecting raw data is the core challenge addressed by FedProf.

The FedProf Methodology

FedProf introduces a novel "representation profiling" technique to dynamically assess client data quality. The method is predicated on the theoretical and empirical observation that internal neural network activations tend to follow Gaussian (normal) distributions across various layer types, including convolutional and residual blocks.

Activation distributions in Conv1 and Res. block layers, demonstrating the Gaussian-like nature of internal representations.

The FedProf workflow consists of four primary stages:

  1. Profiling: Clients compute the mean and variance of activations for specific layers (e.g., Conv1 or Res. blocks, as shown in Figure 2) during a local forward pass. This "Representation Profile" (RP) serves as a privacy-preserving statistical summary.
  2. Matching: The central server maintains a baseline profile derived from a small, clean validation set. It calculates the similarity between client profiles and the baseline using a simplified Kullback–Leibler (KL) divergence formula.
  3. Selective Participation: Clients with high divergence (indicating poor data quality or "weird" distributions) are assigned lower selection probabilities for subsequent training rounds.
  4. Privacy Preservation: To ensure privacy, these statistical summaries can be encrypted using Homomorphic Encryption, allowing the server to perform scoring without accessing raw statistics.

Experimental Evaluation and Results

The researchers evaluated FedProf using diverse datasets, including GasTurbine (IoT), EMNIST (mobile), and CIFAR-10 (cross-silo). To simulate real-world adversity, client data was intentionally corrupted with Gaussian noise, salt-and-pepper noise, blurring, and extreme class imbalances.

FedProf demonstrated significant improvements over state-of-the-art baselines like FedProx and AFL:

  • Convergence Speed: Reached target accuracy up to 2.4x faster than standard baselines.
  • Model Accuracy: Achieved a final accuracy improvement of up to 6.8%.
  • Efficiency: Reduced communication rounds by up to 63% and decreased on-device energy consumption by 37% in large-scale scenarios.

Conclusion

By leveraging the statistical properties of neural representations, FedProf effectively filters out low-quality data contributions in Federated Learning. This approach not only accelerates training but also enhances the robustness and efficiency of the resulting global model. Future work will focus on balancing these performance gains with fairness considerations to ensure that unique but valid data distributions are not inadvertently marginalized.

Summary updated

BibTeX
@inproceedings{wu2022fedprof,
  author = {Wentai Wu and Ligang He and Weiwei Lin and Rui Mao and Chenlin Huang and Wei Song},
  title = {{FedProf: Optimizing Federated Learning with Dynamic Data Profiling}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=jE_ipyh20rb},
}
Figure