Prioritized Multi-Criteria Federated Learning

Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, Antonio FerraraarXiv 2020 · arXiv

TL;DR

This paper introduces a prioritized multi-criteria approach to Federated Learning (FL) aggregation, moving beyond standard methods that weight client updates solely by dataset size. By incorporating qualitative metrics like label diversity and model divergence into a prioritized scoring function, the authors demonstrate significant improvements in convergence speed and model quality, particularly in realistic, non-IID data scenarios.

What Problem Does This Paper Address?

Standard Federated Learning algorithms, most notably FederatedAveraging (FedAvg), aggregate local model updates from clients by weighting them based on the number of samples each client holds. This "quantity over quality" approach assumes that all data points are equally valuable.

In real-world deployments, however, data is often non-IID (not Independent and Identically Distributed). One client might have thousands of blurry, redundant images, while another has a handful of high-quality, diverse samples. FedAvg would heavily favor the former, potentially slowing down convergence or biasing the global model. This paper addresses the need for a more nuanced aggregation strategy that accounts for the qualitative characteristics of client data and local models without compromising user privacy.

Methods

The authors propose Prioritized Multi-Criteria Federated Learning, which modifies the aggregation step by assigning a weight ($w_a$) to each client based on a suite of measurable criteria.

  1. Criteria Identification: They define four classes of criteria:
  2. Data Quality: e.g., dataset size, diversity of training samples.
  3. Model Quality: e.g., how much a local model's gradients diverge from the global model.
  4. Device Trustworthiness: e.g., reliability or connection stability.
  5. Commercial/Targeted: e.g., specific user demographics like age or job.
  6. Prioritized Score Function: To combine multiple criteria, they use a specific mathematical score function: $$f_m(c) = \sum_{i=1}^m \prod_{j=1}^i c_j$$ This function is prioritized: a higher-priority criterion must be satisfied for lower-priority ones to contribute to the final score. If a top-priority criterion is zero (unfulfilled), the entire score becomes zero, regardless of other metrics.

  7. Aggregation: The central server collects model updates ($\Theta_a$) and the criteria measurements ($c_a$) from clients. It computes weights $w_a$ using the score function and produces the new global model.

Information flow in Federated Learning (d) compared to other architectures.

Datasets

The approach was evaluated on two standard benchmarks:

  • MNIST: Handwritten digit recognition. The authors tested both a standard IID split and a challenging Non-IID split where each client only holds samples from at most two digit classes.
  • CelebA: A large-scale face attributes dataset. This was treated as a "user-aware" scenario where each celebrity's photos represent a unique client's gallery, making it inherently non-IID. The task was binary classification (smiling vs. non-smiling).

Key Insights and Ideas

  • IID vs. Non-IID Sensitivity: In perfectly balanced (IID) scenarios, sophisticated weighting offers little benefit because every client is statistically a "mirror" of the others. The proposed method shines in Non-IID settings, which are the norm in real-world applications.
  • The Power of Class Balance: For the CelebA dataset, prioritizing Class Balance (CB)—how evenly distributed the "smiling" vs "non-smiling" labels are—proved far more effective than weighting by dataset size alone.
  • Divergence as a Double-Edged Sword: Penalizing local models that diverge significantly from the global model (the MW criterion) helps stabilize early training but can hinder the final "fine-tuning" stages where unique local features might be valuable.
  • Privacy-Preserving Metadata: The criteria used (like image sharpness or label counts) are "non-sensitive" metadata. Sharing these allows for smarter aggregation without exposing the raw private data itself.

Key Results

  • CelebA Performance: Using the Class Balance (CB) criterion alone allowed the model to reach 85% accuracy in less than half the rounds required by the standard FedAvg baseline.
  • Optimal Combinations: The best-performing configuration for CelebA was a prioritized sequence of Dataset Size $\succ$ Class Balance $\succ$ Image Sharpness, which gained up to 12 communication rounds over the baseline.
  • MNIST Non-IID: The combination of Model Divergence $\succ$ Label Diversity $\succ$ Dataset Size showed a performance boost for high accuracy targets (90-95%), reaching goals up to 4.7 rounds faster than the baseline.

Limitations and Future Work

  • Domain Dependency: There is no universal "best" set of criteria. The efficacy of a specific criterion depends heavily on the dataset and the task.
  • Honesty Assumption: The current framework assumes clients are "honest" and report their criteria measurements truthfully. Future work could investigate how to handle malicious clients who might lie about their data quality to gain more influence.
  • Dynamic Priorities: The authors suggest developing self-adaptive models that can automatically re-rank the priority of criteria during the training process as the global model matures.
  • Fairness and Bias: Using user-specific criteria could potentially lead to biased models (e.g., favoring one demographic over another). The authors identify integrating fairness-aware metrics as a critical next step.

Summary updated

Abstract

In Machine Learning scenarios, privacy is a crucial concern when models have to be trained with private data coming from users of a service, such as a recommender system, a location-based mobile service, a mobile phone text messaging service providing next word prediction, or a face image classification system. The main issue is that, often, data are collected, transferred, and processed by third parties. These transactions violate new regulations, such as GDPR. Furthermore, users usually are not willing to share private data such as their visited locations, the text messages they wrote, or the photo they took with a third party. On the other hand, users appreciate services that work based on their behaviors and preferences. In order to address these issues, Federated Learning (FL) has been recently proposed as a means to build ML models based on private datasets distributed over a large number of clients, while preventing data leakage. A federation of users is asked to train a same global model on their private data, while a central coordinating server receives locally computed updates by clients and aggregate them to obtain a better global model, without the need to use clients' actual data. In this work, we extend the FL approach by pushing forward the state-of-the-art approaches in the aggregation step of FL, which we deem crucial for building a high-quality global model. Specifically, we propose an approach that takes into account a suite of client-specific criteria that constitute the basis for assigning a score to each client based on a priority of criteria defined by the service provider. Extensive experiments on two publicly available datasets indicate the merits of the proposed approach compared to standard FL baseline.
BibTeX
@misc{anelli2020prioritized,
  author = {Vito Walter Anelli and Yashar Deldjoo and Tommaso Di Noia and Antonio Ferrara},
  title = {{Prioritized Multi-Criteria Federated Learning}},
  year = {2020},
  eprint = {2007.08893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.08893},
}
Figure