Pretraining Client Selection Algorithm Based on a Data Distribution Evaluation Model in Federated Learning

Chang Xu, Hong Liu, Kexin Li, W. Feng, Qi WeiIEEE Access 2024 · DOI

TL;DR

This paper introduces DDEM (Data Distribution Evaluation Model), a pre-training algorithm that selects the most valuable participants in a Federated Learning (FL) network before training even begins. By prioritizing clients with rare data classes and accounting for diminishing returns in data volume, DDEM significantly boosts model accuracy and convergence speed in challenging "Non-IID" scenarios where data is unevenly distributed across users.

What Problem Does This Paper Address?

In Federated Learning, a central server trains a model using data stored locally on thousands of user devices (clients) without ever seeing the raw data itself. This is great for privacy, but it hits a major snag called Non-IID data.

In the real world, data isn't "Independently and Identically Distributed." One user might have 1,000 photos of cats, while another has three photos of airplanes. If the server randomly picks clients to participate in a training round, it might end up with a biased model that is great at recognizing cats but fails at everything else. Existing solutions often try to fix this during training by looking at model gradients, but this is computationally expensive, slow, and can leak privacy. The authors set out to find a way to pick the "best" clients before the heavy lifting of training starts.

Methods

The authors propose the Client Selection Algorithm Based on a Data Distribution Evaluation Model (DDEM). The core of the method lies in two specific metrics:

  1. Data Volume and Increment ($\phi_{e\_volume}$): Instead of just counting total samples, the model looks at the volume per class. It uses an integral function with a "marginal diminishing effect"—meaning the first few samples of a class are worth much more to the model's "intelligence" than the 10,000th sample.
  2. Global Data Balance ($\phi_{c\_rareness}$): This is the "secret sauce." The server looks at the distribution of classes across the entire pool. If a specific class (e.g., "Airplanes") is rare across the whole network, clients who possess that rare data are given a much higher evaluation score.

The Workflow: * Step 1: The server collects metadata about data distributions (not the data itself) from the client pool. * Step 2: The server defines an evaluation function based on global rarity. * Step 3: Clients calculate their own "value score" locally and send it back. * Step 4: The server selects the top $K$ clients and starts the FL task.

Datasets

The researchers validated their approach using three distinct datasets, simulating various degrees of Non-IID imbalance using Dirichlet distributions:

  • CIFAR-10: A standard image classification benchmark (10 categories of objects/animals).
  • Fashion-MNIST (FMNIST): Grayscale images of clothing items.
  • DEAP: A complex, real-world dataset for EEG-based emotion recognition. This is particularly challenging because physiological data is naturally highly imbalanced and unique to individuals.

Key Insights and Ideas

  • Rarity is Value: The most important insight is that a client's value isn't just about how much data they have, but how much unique information they contribute to the global "knowledge" of the model.
  • Pre-training Selection is Efficient: By selecting clients once at the start, the system avoids the massive communication overhead of evaluating clients in every single training round.
  • Global vs. Local Balance: A client might have a very "unbalanced" local dataset (only one class), but if that class is what the global model is missing, that client is actually a high-priority candidate. DDEM balances these two perspectives.

Key Results

The DDEM algorithm consistently outperformed standard baselines like Random Selection (RS) and Quantity-Based Selection (QBS):

  • Accuracy Boost: On CIFAR-10, DDEM improved average precision by 5.99% over the best baseline. On the highly imbalanced DEAP dataset, it saw a 5.30% jump.
  • Faster Convergence: In head-to-head tests, DDEM reached 60% accuracy in just 90 rounds, whereas the next best method (DICE) took 170 rounds, and others took up to 390 rounds.
  • Robustness: The performance gap between DDEM and other methods widened as the data became more "Non-IID," proving its effectiveness in messy, real-world data environments.

Limitations and Future Work

While the paper demonstrates strong results, a few areas remain open for exploration:

  • Honesty Assumptions: The model assumes clients report their data distributions accurately. In a competitive or malicious environment, a client might lie about having "rare" data to get selected (and paid).
  • Static Selection: The current algorithm selects clients once before training. If a client's data changes significantly during a long-running training process, the initial selection might become sub-optimal.
  • Computational Overhead: While lower than per-round evaluation, there is still an initial "pre-training" phase where the server must coordinate with the entire pool to establish the global distribution.

Overall, DDEM provides a mathematically grounded, privacy-preserving way to make Federated Learning smarter and faster by simply picking the right team for the job before the whistle blows.

Summary updated

Abstract

Federated Learning (FL) allows task initiators (servers) to utilize data from task participants (clients) to train machine learning models while protecting data privacy. However, in the FL system, when the client data are non-independently identically distributed (Non-IID), appropriate metrics are chosen to accurately evaluate the quality of the client data, accordingly to select a reasonable subset of clients, and thus ensure the accuracy of the FL aggregation model. In this paper, based on the experimental results, a data distribution evaluation model is proposed, which is based on two metrics: the volume of client data and its increment and the balance of global client data. This data distribution evaluation model enables more accurate evaluation of clients with Non-IID characteristics. Based on this evaluation model, this paper further proposes an FL client subset selection algorithm. This algorithm accurately evaluates the data value of each client, enabling the server to select the most valuable subset of clients before FL training, thus improving the accuracy of the federated learning aggregation model in scenarios with Non-IID client data. When training the FL aggregation model using the proposed method on datasets composed of CIFAR-10, Fashion-MNIST, and DEAP distributions, compared to the optimal baseline, the average precision scores increased by 5.99%, 4.79%, and 4.29% respectively. The improvement in accuracy is more pronounced in scenarios with Non-IID data, such as in the DEAP dataset distribution with the highest Non-IID degree, where the accuracy increased by 5.30% compared to the optimal baseline.
BibTeX
@inproceedings{xu2024pretraining,
  author = {Chang Xu and Hong Liu and Kexin Li and W. Feng and Qi Wei},
  title = {{Pretraining Client Selection Algorithm Based on a Data Distribution Evaluation Model in Federated Learning}},
  booktitle = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3396279},
}
Figure