TL;DR
Traditional federated learning fairness metrics like accuracy parity inadvertently penalize clients with high-quality data by forcing them to match the performance of noisy or low-quality clients. This paper introduces Fairness via Agent-Awareness (FAA), a new metric based on equalizing excess risk relative to each client's best possible local model, and FOCUS, an EM-based soft-clustering algorithm that groups similar clients to optimize this new notion of fairness without sacrificing accuracy.
What Problem Does This Paper Address?
Federated Learning (FL) allows multiple decentralized clients (agents) to collaboratively train a shared machine learning model without centralizing their raw data. However, real-world data is highly heterogeneous—different clients possess data of vastly different qualities, noise levels, and underlying distributions.
Existing "fair" FL frameworks typically pursue accuracy parity, attempting to minimize the performance discrepancy across all clients. The authors argue that this definition is fundamentally flawed in heterogeneous settings. If a well-behaved hospital invests heavily in collecting pristine, high-resolution medical images, forcing its model performance to match that of a hospital contributing low-quality, poorly labeled data is inherently unfair. This "blind parity" penalizes high-quality contributors and discourages them from participating in the network. The challenge is to define and enforce a fairness metric that respects the inherent constraints and quality of each client's local data.
Methods
The paper tackles this problem with a two-pronged approach: a new fairness metric and a novel optimization algorithm.
- Fairness via Agent-Awareness (FAA): Instead of equalizing absolute accuracy, FAA equalizes excess risk. Excess risk is defined as a client's current loss minus their Bayes optimal error (the absolute lowest error theoretically possible given their local data distribution). By minimizing the maximum difference in excess risk between any pair of clients, FAA ensures an "equity of gains" rather than an equity of absolute performance.
- FOCUS (Fairness via Agent-Clustering) Algorithm: To optimize for FAA, the authors propose an Expectation-Maximization (EM) style soft-clustering framework.
- E-step: The central server estimates the probability ($\Pi$) of each agent belonging to each of $M$ clusters based on how well the cluster models perform on the agent's local data.
- M-step: Agents initialize local models with their assigned cluster weights, perform local SGD updates, and send them back to the server. The server then aggregates these updates using the soft-clustering weights to compute new cluster models.
During inference, an unseen or participating agent makes predictions using a weighted ensemble of all cluster models based on their soft-clustering coefficients.
Datasets
The authors evaluated their framework across four distinct heterogeneous data settings:
- Synthetic Data: A linear regression setup generated from Gaussian distributions, featuring a single extreme outlier client to rigorously test theoretical bounds.
- Rotated MNIST: The classic handwritten digit dataset, split into 10 client subsets where specific subsets are rotated by 90 or 180 degrees to simulate distinct ground-truth distribution clusters.
- Rotated CIFAR: The CIFAR-10 image dataset, similarly manipulated with 180-degree rotations to create two distinct ground-truth client clusters.
- Yelp / IMDb: A text sentiment classification setup where 7 clients hold Yelp restaurant reviews and 3 clients hold IMDb movie reviews, naturally forming a heterogeneous text environment.
Key Insights and Ideas
- Equity of Gains over Parity: True fairness in collaborative learning shouldn't mean everyone gets the same final score; it means everyone benefits equitably relative to what their data allows.
- Isolating Outliers Protects the Group: In standard FedAvg, a single noisy outlier client can corrupt the global model for everyone. By leveraging clustering, FOCUS naturally isolates anomalous or noisy distributions into their own clusters, protecting benign clients while still providing a tailored model for the outlier.
- The Power of Soft Clustering: Unlike hard clustering (where an agent belongs to exactly one cluster), soft clustering allows clients to benefit from shared knowledge across multiple cluster models simultaneously. This is particularly effective when client distributions overlap or are not perfectly separable.
Key Results
The empirical evaluation demonstrates that FOCUS successfully balances high accuracy with superior fairness:
- Higher Fairness, Better Accuracy: On Rotated MNIST, FOCUS achieved an FAA score of 0.094 (lower is fairer) compared to FedAvg's 0.363, while simultaneously improving average test accuracy (95.3% vs 92.9%). Traditional fair methods like q-FFL dropped overall accuracy significantly (down to 68.5% at $q=10$) to force parity.
- Flattening Excess Risk: As illustrated in the paper's client-by-client risk breakdown, FOCUS successfully drives down the massive excess risk spikes seen in outlier clients under standard FedAvg, distributing the optimization gains much more equitably.
- Scalability and Faster Convergence: When scaled up to a massive 100-client simulation on Rotated MNIST, FOCUS converged faster and achieved a lower test loss than FedAvg, q-FFL, AFL, and Ditto.
Limitations and Future Work
- Choosing the Number of Clusters ($M$): The algorithm requires the user to predefine the number of clusters $M$. While the authors show that FOCUS is robust if $M$ is larger than the true number of underlying distributions (superfluous clusters simply see their weights drop to zero), setting $M$ too small can force heterogeneous agents into the same cluster.
- Inference Latency: Because inference requires running an ensemble prediction across all $M$ cluster models, it introduces a slight computational overhead (reported as roughly 0.17 seconds of extra latency on MNIST compared to FedAvg).
- Theoretical Assumptions: The convergence proofs rely on the assumption that the underlying local data distributions are reasonably separable into distinct clusters, which may not hold in extremely chaotic or continuous non-IID data environments.
Summary updated