Diversity-Aware Client Selection for Communication-Efficient Federated Learning

Max Halleryd, Sima Sinaei, Viktor Valadi, Mina Alibeigi, Henrik Abrahamsson, Salman ToorarXiv 2026 · arXiv

TL;DR

This paper demonstrates that in Federated Learning, communication efficiency and model quality don't have to be a zero-sum game. By using diversity-aware client selection—specifically the "Power-of-Choice" strategy—researchers achieved a 6.46x reduction in communication overhead while maintaining competitive accuracy, even under challenging non-IID (heterogeneous) data conditions.

What Problem Does This Paper Address?

Federated Learning (FL) allows models to be trained across distributed devices without moving raw data, which is a win for privacy. However, it faces two massive hurdles:

  1. Communication Bottlenecks: Constantly sending model updates back and forth is expensive and slow, especially for edge devices or vehicles.
  2. Data Heterogeneity (Non-IID): Real-world data isn't distributed evenly. If a selection strategy accidentally ignores clients with unique data (e.g., a specific class of images), the final global model will be biased and perform poorly.

Most research focuses on how fast a model converges or its final accuracy. This paper asks a deeper question: Who are we picking to train, and does that choice lead to a diverse, representative model?

Methods

The authors evaluate three distinct client selection strategies against a "full participation" baseline:

  • Power-of-Choice: Prioritizes clients with the highest local loss. The intuition is that clients "struggling" the most likely have the most to teach the global model.
  • Fisher Information: Ranks clients based on the sensitivity of their gradients. It picks clients whose data would cause the most significant (and presumably informative) changes to the model parameters.
  • Centered Kernel Alignment (CKA): Measures the similarity between a client's internal data representations and the global model, prioritizing those that are most different.

To measure "diversity," the authors borrowed tools from economics and information theory:

  • Gini Coefficient: Used to measure participation inequality. A lower Gini means clients are being selected more uniformly.
  • KL Divergence: Measures how much the label distribution of the selected clients deviates from a perfectly balanced (uniform) distribution.

Datasets

The study uses two benchmarks partitioned to simulate "label skew" (where different clients have different classes of data):

  • MNIST: The classic handwritten digit dataset, used here as a controlled environment to isolate the effects of selection.
  • ImageNet50: A more complex "stress test" consisting of 50 classes from the ImageNet-1K dataset. This provides the high-dimensional complexity needed to see where selection strategies break down.

Key Insights and Ideas

  • Diversity is a Leading Indicator: The researchers found that diversity metrics (Gini and KL) aren't just descriptive; they are predictive. Strategies that show lower inequality and better class coverage early in training consistently result in better final generalization.
  • Loss-Guided Selection is a Natural Balancer: Power-of-Choice naturally "rotates" through underrepresented clients. Because these clients have data the model hasn't mastered yet, their local loss remains high, ensuring they get picked eventually.
  • The CKA Pitfall: Interestingly, CKA-based selection performed poorly. Its similarity-based approach led to biased participation, proving that "looking for difference" in representations isn't always as effective as looking for high error (loss).
  • Fairness vs. Size: In the ImageNet50 tests, clients with the smallest datasets were often selected most frequently by Power-of-Choice because their unique data led to higher training losses. This suggests that loss-based selection can actually favor "small but important" data sources.
Fig. 4: Gini coefficient (ImageNet50) over 350 rounds, where 3 clients are selected in each round.

Key Results

The core finding is that restricting participation to just 15% of clients (3 out of 20) per round yields massive savings with manageable accuracy trade-offs:

  • Communication Savings: A reduction of 244 MB for MNIST and a staggering 659 GB for ImageNet50 compared to full participation.
  • Accuracy: On ImageNet50, the Power-of-Choice strategy achieved 53.22% test accuracy, compared to the baseline's 59.77%. While there is a gap, it is remarkably narrow considering the 85% reduction in client participation and the lack of additional regularization.
  • Stability: As seen in the KL divergence plots, diversity-aware strategies like Fisher and Power-of-Choice eventually converge to very low divergence, indicating they successfully "see" a balanced representation of all classes over time.
Fig. 5: KL divergence to the uniform distribution (ImageNet50); 3 clients selected per round.

Limitations and Future Work

The authors acknowledge that their study used a relatively small pool of 20 clients and a fixed participation rate. In massive, real-world deployments (like millions of smartphones), the dynamics might shift.

Future research directions include:

  • Adaptive Selection: Developing mechanisms that automatically adjust the participation rate or strategy as the model matures.
  • Stronger Non-IID Conditions: Testing these strategies against even more extreme data imbalances and more complex, real-world datasets.
  • Explicit Data Coverage: Creating selection algorithms that explicitly account for class coverage rather than relying on loss as a proxy.

Summary updated

Abstract

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, making it attractive for privacy-sensitive and bandwidth-constrained applications. A practical challenge in deploying FL is reducing communication overhead while maintaining model performance under heterogeneous, non-IID data distributions. Selecting which subset of clients participates in each training round offers a principled way to address both challenges simultaneously, yet its impact on data diversity and representativeness remains underexplored. In this paper, we investigate three client selection strategies, Power-of-Choice, Fisher information, and Centered Kernel Alignment, evaluating them not only by accuracy and convergence, but through the lens of participation diversity using the Gini coefficient and KL divergence. Under non-IID conditions, restricting participation to 15% of clients per round achieves significant communication savings, and when selection promotes diverse and balanced contributions, it also improves generalization over biased alternatives. This work provides a systematic and reproducible analysis demonstrating that diversity-focused client selection renders communication efficiency and model quality complementary rather than competing objectives.
BibTeX
@misc{halleryd2026diversity,
  author = {Max Halleryd and Sima Sinaei and Viktor Valadi and Mina Alibeigi and Henrik Abrahamsson and Salman Toor},
  title = {{Diversity-Aware Client Selection for Communication-Efficient Federated Learning}},
  year = {2026},
  eprint = {2611.09278},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09278},
}
Figure