This paper addresses a critical but often overlooked issue in decentralized machine learning: the "rich get richer" problem of model accuracy. Standard Federated Learning (FL) focuses on maximizing average accuracy, which often leaves "outlier" devices with poor performance. To counter this, the authors introduce $q$-Fair Federated Learning ($q$-FFL), a flexible optimization objective that allows practitioners to trade off average performance for a fairer, more uniform distribution of accuracy across all devices in a network.
The Challenge of Heterogeneity in Federated Networks
Federated Learning involves training models across thousands or millions of remote devices, such as smartphones or IoT sensors. Because these devices possess vastly different data sizes and distributions—a property known as heterogeneity—a model that performs well on average might perform terribly for a significant subset of users.
Existing "fairness" research in machine learning usually focuses on protecting specific demographic attributes. In the federated setting, however, the "groups" are the devices themselves. Prior attempts to solve this, such as Agnostic Federated Learning (AFL), were often too rigid. They focused exclusively on the single worst-performing device and struggled to scale to massive networks. The authors ask a fundamental question: How can we efficiently encourage a more uniform distribution of performance across all devices?
Proposed Method: $q$-FFL and $q$-FedAvg
To address this, the authors propose a new objective function and a scalable algorithm to solve it:
1. The $q$-FFL Objective: Inspired by fair resource allocation in wireless networks, the authors propose reweighting the aggregate loss function. The objective is defined as: $$\min_w f_q(w) = \sum_{k=1}^m \frac{p_k}{q+1} F_k^{q+1}(w)$$ Here, $q$ is a tunable parameter that acts as a "knob" for fairness. When $q=0$, the objective reduces to standard FedAvg, prioritizing global average accuracy. As $q$ increases, the objective gives exponentially more weight to devices with higher local loss—the ones the model is currently failing. High $q$ values enforce strict minimax fairness.
2. $q$-FedAvg Algorithm: To solve this objective at scale, the authors developed $q$-FedAvg, a communication-efficient version of the standard Federated Averaging algorithm. Unlike previous minimax approaches that require full device participation every round, $q$-FedAvg works with partial device sampling, making it viable for real-world, large-scale deployments.
3. Dynamic Step-Sizing: A key technical contribution is a method to automatically adjust step sizes. By estimating the local Lipschitz constant of the gradient, the algorithm avoids the need for exhaustive hyperparameter tuning for every different value of $q$.
Key Insights and Results
The researchers evaluated their approach on a diverse suite of federated benchmarks, including Synthetic data, Vehicle sensor networks, Sent140 (Twitter sentiment), Shakespeare (next-character prediction), and image datasets like Omniglot and Fashion MNIST.
A core insight of the paper is defining fairness not as parity between specific groups, but as the uniformity of the accuracy distribution. A "fair" model minimizes the performance gap between the best and worst devices. Furthermore, the authors discovered that $q$-FFL synergizes well with meta-learning (e.g., MAML), producing fairer model initializations that adapt more consistently across new tasks.
Empirical results strongly validate the approach:
- Reduced Variance: On average, $q$-FFL reduced the variance of accuracies across devices by 45% while maintaining nearly the same overall average accuracy.
- Boosting the Bottom 10%: In the "Vehicle" dataset, $q$-FFL improved the accuracy of the worst-performing 10% of devices from 43.0% to 69.9%.
- Superior to Baselines: $q$-FFL outperformed Agnostic FL (AFL) in both final accuracy on struggling devices and convergence speed.
- Scalability: The algorithm successfully handled datasets with over 1,000 devices, a significant jump from the 2–3 devices used in prior fairness studies.
Limitations and Future Work
While highly effective, the approach has limitations. The authors acknowledge that very high values of $q$ can theoretically make the optimization problem more difficult, potentially slowing down convergence. Additionally, in cases of extreme data diversity, the "local update" heuristic in $q$-FedAvg can sometimes be less efficient than global gradient descent ($q$-FedSGD) because local models might drift too far apart. Finally, while the paper provides a robust framework, the "optimal" $q$ remains application-dependent, suggesting that future work could involve automated methods to find the best $q$ based on specific fairness-accuracy constraints.
Summary updated