Towards Effective Device-Aware Federated Learning

Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, Antonio FerraraarXiv 2019 · arXiv

TL;DR

Standard Federated Learning (FL) typically weights client contributions based solely on local dataset size, ignoring data quality and device characteristics. This paper introduces a prioritized, multi-criteria aggregation framework that dynamically adjusts the importance of factors like label diversity and model divergence during training, significantly speeding up convergence in heterogeneous environments.

What Problem Does This Paper Address?

Federated Learning allows multiple clients (like smartphones or IoT devices) to collaboratively train a machine learning model without sharing their raw, private data. A central server coordinates this by aggregating local model updates.

The current gold standard for this aggregation is FedAvg. However, FedAvg has a significant blind spot: it assumes that the only thing that matters is the quantity of data a client has. It weights updates proportionally to local dataset size.

In the real world, data is "non-IID" (not independently and identically distributed). One client might have a massive dataset that is entirely one-sided (low diversity), while another might have a small but highly representative dataset. Furthermore, some local models might diverge wildly from the global goal, potentially harming the collective model. The authors argue that relying solely on dataset size is too simplistic and leads to inefficient training.

Methods

The authors propose a "device-aware" FL approach built on three main pillars:

  1. Multi-Criteria Evaluation: Instead of just looking at dataset size (DS), they introduce two new qualitative criteria:
  2. Label Diversity (Ld): How many different classes are represented in the local data?
  3. Model Divergence (Md): How much does the local model's parameters differ from the current global model? (Updates that diverge too much are penalized).
  4. Prioritized Aggregation: Borrowing a technique from Information Retrieval, they use a prioritized multi-criteria aggregation operator. This allows a domain expert to define a hierarchy of importance. For example, if "Model Divergence" is the top priority, a client with a huge dataset (high DS) but a wildly diverging model (low Md) will have its contribution heavily penalized. The fulfillment of a lower-priority criterion cannot fully compensate for the failure of a higher-priority one.
  5. Online Parameter Adjustment: Because the "best" priority order might not be obvious or might change during training, the authors developed an online adjustment mechanism. The server monitors global accuracy. If accuracy drops in a round, the system performs a local search—backtracking to the previous state and trying a different permutation of criteria priorities—to find the ordering that maximizes performance.
Fig. 1. An illustration of the online parameter adjustment for the aggregation operator.
Figure 1: The training process dynamically switches between different parameterizations (A, B, C) to maintain an upward trajectory in accuracy.

Datasets

The researchers evaluated their approach using the FEMNIST dataset. Unlike the standard MNIST (handwritten digits), FEMNIST is partitioned by the actual writer of the characters, making it a much more realistic, non-IID benchmark for federated settings. They used a subsampled version involving 371 clients performing a digit/character classification task using a Convolutional Neural Network (CNN).

Key Insights and Ideas

  • Quality over Quantity: The most important takeaway is that data quality (diversity and alignment with the global goal) is often more important than raw volume in FL.
  • The Power of Priority: Using a prioritized operator is more effective than simple weighted averaging because it allows for "bottleneck" criteria. If a model update is too divergent, no amount of extra data should make it highly influential.
  • Dynamic Adaptation: Training isn't static. The "Final" model, which used the online adjustment strategy, consistently outperformed static priority models, suggesting that the optimal weighting strategy evolves as the global model matures.

Key Results

The authors measured how many communication rounds were needed to reach target accuracies of 75% and 80% across different percentages of the client population.

  • Superior Efficiency: To reach 75% accuracy for 75% of the devices, the standard FedAvg (base DS) took 801 rounds. Their best dynamic model (Final) reached the same goal in just 494 rounds—a nearly 40% improvement in communication efficiency.
  • Individual Criteria Impact: When tested in isolation, Label Diversity (Ld) was a much stronger predictor of a "good" update than dataset size for reaching high-accuracy thresholds. Ld alone reached the 75% target for the "High" device category in 405 rounds, compared to 552.5 for the baseline.
  • Convergence at Higher Thresholds: The baseline FedAvg often failed to reach the 80% accuracy target for a high percentage of devices within the 1,000-round limit, whereas the proposed prioritized methods succeeded.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Criteria Selection: While DS, Ld, and Md proved effective, there are likely many other domain-specific criteria (e.g., device battery life, connection stability, or user demographics) that could be integrated.
  • Aggregation Operators: The study focused on one specific prioritized operator; other fuzzy logic or Choquet-based models might offer different benefits.
  • Task Diversity: The current work focused on image classification. Future work should test this framework on other tasks, such as federated recommendation systems.

Summary updated

Abstract

With the wealth of information produced by social networks, smartphones, medical or financial applications, speculations have been raised about the sensitivity of such data in terms of users' personal privacy and data security. To address the above issues, Federated Learning (FL) has been recently proposed as a means to leave data and computational resources distributed over a large number of nodes (clients) where a central coordinating server aggregates only locally computed updates without knowing the original data. In this work, we extend the FL framework by pushing forward the state the art in the field on several dimensions: (i) unlike the original FedAvg approach relying solely on single criteria (i.e., local dataset size), a suite of domain- and client-specific criteria constitute the basis to compute each local client's contribution, (ii) the multi-criteria contribution of each device is computed in a prioritized fashion by leveraging a priority-aware aggregation operator used in the field of information retrieval, and (iii) a mechanism is proposed for online-adjustment of the aggregation operator parameters via a local search strategy with backtracking. Extensive experiments on a publicly available dataset indicate the merits of the proposed approach compared to standard FedAvg baseline.
BibTeX
@misc{anelli2019towards,
  author = {Vito Walter Anelli and Yashar Deldjoo and Tommaso Di Noia and Antonio Ferrara},
  title = {{Towards Effective Device-Aware Federated Learning}},
  year = {2019},
  eprint = {1908.07420},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.07420},
}
Figure