Fair Concurrent Training of Multiple Models in Federated Learning

Marie Siew, Haoran Zhang, Jong-Ik Park, Yuezhou Liu, Yichen Ruan, Lili Su, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong

Abstract

Federated Learning (FL) typically involves a group of clients collaborating to train a single global model. However, many real-world applications require the simultaneous training of multiple distinct models across a shared pool of heterogeneous clients. This paper introduces the Multiple-Model Federated Learning (MMFL) framework and addresses the challenges of task difficulty imbalance and client preference heterogeneity. The authors propose FedFairMMFL, a difficulty-aware allocation algorithm, and a novel Max-Min Fair Auction mechanism to ensure fair performance and participation across all tasks.

Introduction

In standard FL, the focus is on optimizing a single objective. In practice, a network of devices (e.g., smartphones or IoT sensors) might need to support various tasks simultaneously, such as heart rate monitoring alongside disease detection. This "Multiple-Model" setting introduces two primary hurdles:

  1. Heterogeneous Task Difficulty: Different models (e.g., complex image recognition vs. simple sensor data) require different amounts of data and computation to reach target accuracy.
  2. Client Preferences: Clients may be biased towards certain tasks due to personal interest or resource costs, leading to uneven training progress across the model suite.

Methodology: The FedFairMMFL Framework

To address these challenges, the authors propose a two-stage approach encompassing client recruitment and fair task allocation.

Overview of the MMFL framework, showing the client recruitment auction and the subsequent training process.

Difficulty-Aware Allocation (FedFairMMFL)

The core of the training process is the FedFairMMFL algorithm. Instead of uniform client distribution, it employs $\alpha$-fairness to dynamically allocate clients based on real-time task performance. By monitoring the current loss of each task, the algorithm assigns a higher probability of client allocation to the "harder" or worst-performing tasks. This ensures that training effort is concentrated where it is most needed, preventing difficult tasks from being neglected.

Algorithm 1: The FedFairMMFL training process.

Max-Min Fair Auction

To manage client willingness and budget constraints, a truthful auction mechanism is integrated. This auction aims to maximize the "minimum take-up rate" across all tasks. By providing financial or resource-based incentives, the server ensures that even unpopular tasks receive sufficient client participation to progress.

Experimental Setup

The framework was evaluated using four computer vision benchmarks: MNIST, Fashion-MNIST, CIFAR-10, and EMNIST. These datasets represent a spectrum of difficulty levels. The experiments simulated non-i.i.d. data distributions, where each client holds only a subset of classes, reflecting the fragmented nature of real-world data.

Key Results and Insights

  • Fairness and Accuracy: FedFairMMFL significantly improves the accuracy of the "worst-case" models without substantial degradation of average performance. By using loss as a proxy for difficulty, the system adapts to model requirements without prior knowledge of their architectures.
  • Convergence and Variance: Difficult tasks reached target accuracy faster under FedFairMMFL compared to round-robin baselines. Furthermore, the variance in accuracy across different tasks was reduced, leading to more uniform outcomes.
  • Auction Efficiency: The Max-Min Fair Auction proved more effective than standard budget-fair auctions in balancing client distribution, particularly when the server's budget was limited.

Conclusion and Future Work

This work highlights the importance of fairness in multi-task federated environments. While FedFairMMFL successfully balances task performance, future research could address client-side resource constraints—such as allowing powerful clients to train multiple models per round—and explore the trade-offs between task-level fairness and client-level fairness.

Summary updated

BibTeX
@article{siew2025fair,
  author = {Marie Siew and Haoran Zhang and Jong-Ik Park and Yuezhou Liu and Yichen Ruan and Lili Su and Stratis Ioannidis and Edmund Yeh and Carlee Joe-Wong},
  title = {{Fair Concurrent Training of Multiple Models in Federated Learning}},
  journal = {IEEE Trans. Netw.},
  year = {2025},
  url = {https://openreview.net/forum?id=HhZMeCVbZk},
}
Figure