Evaluation Framework For Large-scale Federated Learning

Lifeng Liu, Fengda Zhang, Jun Xiao, Chao WuarXiv 2020 · arXiv

Introduction: The Challenge of Non-IID Data

Federated Learning (FL) offers a transformative approach to machine learning by enabling global model training across decentralized edge devices without compromising data privacy. However, the practical implementation of FL faces a significant hurdle: data heterogeneity. In real-world scenarios, data is rarely Identically and Independently Distributed (IID). Instead, it is characterized by label skew, feature skew, and concept skew across millions of devices.

Until recently, the field lacked a consistent methodology for quantifying "Non-IID-ness" or a standardized benchmark to evaluate how various FL algorithms perform under these messy, decentralized conditions. This paper introduces EFFL (Evaluation Framework for Federated Learning), an open-source solution designed to bring rigorous standardization to large-scale FL research.

The EFFL Framework

EFFL is a modular framework that streamlines the entire FL research pipeline, from data generation to performance analysis. It provides researchers with the tools to simulate realistic edge computing environments and evaluate model robustness systematically.

The EFFL modular architecture, illustrating the layers from basic services to evaluation metrics.

Quantifying Heterogeneity: The NEI Metric

A core innovation of EFFL is the Non-IID Encoder Index (NEI). Rather than relying on qualitative assessments of data messiness, the authors utilize a fixed AutoEncoder to extract features and calculate the statistical distance between training and test distributions. A higher NEI value indicates a greater degree of distribution shift, providing a concrete, quantitative metric to describe task difficulty.

Simulating Real-World Shifts

To mimic the "Wild West" of edge devices, the framework allows for the intentional manipulation of datasets through three primary mechanisms:

  • Covariate Shift: Simulates varying sensor quality by adding noise (e.g., Gaussian) to inputs.
  • Prior Probability Shift: Partitions data so that different nodes possess distinct label distributions (e.g., one node seeing only specific classes).
  • Concept Shift: Redefines labels based on context to test the model's ability to handle environmental variations.

Experimental Evaluation and Key Insights

The researchers utilized modified versions of the MNIST and CIFAR-10 datasets to validate the framework. Their findings highlight several critical aspects of large-scale FL performance:

  1. Scalability and Node Participation: Increasing the number of participating nodes generally improves the training effect. For instance, on the MNIST dataset under Quantity Skew conditions, increasing from 5 to 30 nodes improved accuracy from approximately 92.55% to over 97.21%.
Model accuracy on MNIST and CIFAR-10 across varying node counts and skew types.
  1. Data Quality vs. Quantity: The study introduced parameters to measure the impact of "dirty" or erroneous data. Results indicate that even minor increases in data errors significantly degrade model accuracy, suggesting that robust FL algorithms must prioritize data filtering and noise handling over sheer volume.
  2. Shift Difficulty: The framework demonstrated that Concept Shifts (NEI ~5.3) are significantly more challenging for models to navigate than simple Covariate Shifts (NEI ~3.1) on the CIFAR-10 dataset, reinforcing the need for specialized handling of contextual variations.

Conclusion and Future Directions

EFFL represents a major step toward standardizing the evaluation of Federated Learning. While the current framework provides a robust foundation, future work will focus on simplifying the NEI calculation, exploring more complex partitioning methods that reflect the nuances of real-world data, and expanding support to high-resolution and non-image datasets. By providing a quantitative language for data heterogeneity, EFFL enables the community to develop more resilient and scalable FL solutions.

Summary updated

Abstract

Federated learning is proposed as a machine learning setting to enable distributed edge devices, such as mobile phones, to collaboratively learn a shared prediction model while keeping all the training data on device, which can not only take full advantage of data distributed across millions of nodes to train a good model but also protect data privacy. However, learning in scenario above poses new challenges. In fact, data across a massive number of unreliable devices is likely to be non-IID (identically and independently distributed), which may make the performance of models trained by federated learning unstable. In this paper, we introduce a framework designed for large-scale federated learning which consists of approaches to generating dataset and modular evaluation framework. Firstly, we construct a suite of open-source non-IID datasets by providing three respects including covariate shift, prior probability shift, and concept shift, which are grounded in real-world assumptions. In addition, we design several rigorous evaluation metrics including the number of network nodes, the size of datasets, the number of communication rounds and communication resources etc. Finally, we present an open-source benchmark for large-scale federated learning research.
BibTeX
@misc{liu2020evaluation,
  author = {Lifeng Liu and Fengda Zhang and Jun Xiao and Chao Wu},
  title = {{Evaluation Framework For Large-scale Federated Learning}},
  year = {2020},
  eprint = {2003.01575},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.01575},
}
Figure