FOCUS: Fairness via Agent-Awareness for Federated Learning on Heterogeneous Data

Wenda Chu, Chulin Xie, Boxin Wang, Linyi Li, Lang Yin, Arash Nourian, Han Zhao, Bo Li

TL;DR

Traditional federated learning fairness metrics like accuracy parity inadvertently penalize clients with high-quality data by forcing them to match the performance of noisy or low-quality clients. This paper introduces Fairness via Agent-Awareness (FAA), a new metric based on equalizing excess risk relative to each client's best possible local model, and FOCUS, an EM-based soft-clustering algorithm that groups similar clients to optimize this new notion of fairness without sacrificing accuracy.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple decentralized clients (agents) to collaboratively train a shared machine learning model without centralizing their raw data. However, real-world data is highly heterogeneous—different clients possess data of vastly different qualities, noise levels, and underlying distributions.

Existing "fair" FL frameworks typically pursue accuracy parity, attempting to minimize the performance discrepancy across all clients. The authors argue that this definition is fundamentally flawed in heterogeneous settings. If a well-behaved hospital invests heavily in collecting pristine, high-resolution medical images, forcing its model performance to match that of a hospital contributing low-quality, poorly labeled data is inherently unfair. This "blind parity" penalizes high-quality contributors and discourages them from participating in the network. The challenge is to define and enforce a fairness metric that respects the inherent constraints and quality of each client's local data.

Methods

The paper tackles this problem with a two-pronged approach: a new fairness metric and a novel optimization algorithm.

  1. Fairness via Agent-Awareness (FAA): Instead of equalizing absolute accuracy, FAA equalizes excess risk. Excess risk is defined as a client's current loss minus their Bayes optimal error (the absolute lowest error theoretically possible given their local data distribution). By minimizing the maximum difference in excess risk between any pair of clients, FAA ensures an "equity of gains" rather than an equity of absolute performance.
  2. FOCUS (Fairness via Agent-Clustering) Algorithm: To optimize for FAA, the authors propose an Expectation-Maximization (EM) style soft-clustering framework.
  3. E-step: The central server estimates the probability ($\Pi$) of each agent belonging to each of $M$ clusters based on how well the cluster models perform on the agent's local data.
  4. M-step: Agents initialize local models with their assigned cluster weights, perform local SGD updates, and send them back to the server. The server then aggregates these updates using the soft-clustering weights to compute new cluster models.

During inference, an unseen or participating agent makes predictions using a weighted ensemble of all cluster models based on their soft-clustering coefficients.

Datasets

The authors evaluated their framework across four distinct heterogeneous data settings:

  • Synthetic Data: A linear regression setup generated from Gaussian distributions, featuring a single extreme outlier client to rigorously test theoretical bounds.
  • Rotated MNIST: The classic handwritten digit dataset, split into 10 client subsets where specific subsets are rotated by 90 or 180 degrees to simulate distinct ground-truth distribution clusters.
  • Rotated CIFAR: The CIFAR-10 image dataset, similarly manipulated with 180-degree rotations to create two distinct ground-truth client clusters.
  • Yelp / IMDb: A text sentiment classification setup where 7 clients hold Yelp restaurant reviews and 3 clients hold IMDb movie reviews, naturally forming a heterogeneous text environment.

Key Insights and Ideas

  • Equity of Gains over Parity: True fairness in collaborative learning shouldn't mean everyone gets the same final score; it means everyone benefits equitably relative to what their data allows.
  • Isolating Outliers Protects the Group: In standard FedAvg, a single noisy outlier client can corrupt the global model for everyone. By leveraging clustering, FOCUS naturally isolates anomalous or noisy distributions into their own clusters, protecting benign clients while still providing a tailored model for the outlier.
  • The Power of Soft Clustering: Unlike hard clustering (where an agent belongs to exactly one cluster), soft clustering allows clients to benefit from shared knowledge across multiple cluster models simultaneously. This is particularly effective when client distributions overlap or are not perfectly separable.

Key Results

The empirical evaluation demonstrates that FOCUS successfully balances high accuracy with superior fairness:

  • Higher Fairness, Better Accuracy: On Rotated MNIST, FOCUS achieved an FAA score of 0.094 (lower is fairer) compared to FedAvg's 0.363, while simultaneously improving average test accuracy (95.3% vs 92.9%). Traditional fair methods like q-FFL dropped overall accuracy significantly (down to 68.5% at $q=10$) to force parity.
  • Flattening Excess Risk: As illustrated in the paper's client-by-client risk breakdown, FOCUS successfully drives down the massive excess risk spikes seen in outlier clients under standard FedAvg, distributing the optimization gains much more equitably.
Figure 1: The excess risks of different agents trained with FedAvg and FOCUS on MNIST (a) and Yelp/IMDb text data (b). Ci denotes ith cluster.
  • Scalability and Faster Convergence: When scaled up to a massive 100-client simulation on Rotated MNIST, FOCUS converged faster and achieved a lower test loss than FedAvg, q-FFL, AFL, and Ditto.
Figure 2: Comparison of the convergence rate of different methods on rotated MNIST with 100 clients.

Limitations and Future Work

  • Choosing the Number of Clusters ($M$): The algorithm requires the user to predefine the number of clusters $M$. While the authors show that FOCUS is robust if $M$ is larger than the true number of underlying distributions (superfluous clusters simply see their weights drop to zero), setting $M$ too small can force heterogeneous agents into the same cluster.
  • Inference Latency: Because inference requires running an ensemble prediction across all $M$ cluster models, it introduces a slight computational overhead (reported as roughly 0.17 seconds of extra latency on MNIST compared to FedAvg).
  • Theoretical Assumptions: The convergence proofs rely on the assumption that the underlying local data distributions are reasonably separable into distinct clusters, which may not hold in extremely chaotic or continuous non-IID data environments.

Summary updated

Abstract

Federated learning (FL) allows agents to jointly train a global model without sharing their local data. However, due to the heterogeneous nature of local data, it is challenging to optimize or even define fairness of the trained global model for the agents. For instance, existing work usually considers accuracy equity as fairness for different agents in FL, which is limited, especially under the heterogeneous setting, since it is intuitively "unfair" to enforce agents with high-quality data to achieve similar accuracy to those who contribute low-quality data, which may discourage the agents from participating in FL. In this work, we propose a formal FL fairness definition, fairness via agent-awareness (FAA), which takes different contributions of heterogeneous agents into account. Under FAA, the performance of agents with high-quality data will not be sacrificed just due to the existence of large amounts of agents with low-quality data. In addition, we propose a fair FL training algorithm based on agent clustering (FOCUS) to achieve fairness in FL measured by FAA. Theoretically, we prove the convergence and optimality of FOCUS under mild conditions for linear and general convex loss functions with bounded smoothness. We also prove that FOCUS always achieves higher fairness in terms of FAA compared with standard FedAvg under both linear and general convex loss functions. Empirically, we show that on four FL datasets, including synthetic data, images, and texts, FOCUS achieves significantly higher fairness in terms of FAA while maintaining competitive prediction accuracy compared with FedAvg and state-of-the-art fair FL algorithms.
BibTeX
@inproceedings{chu2023focus,
  author = {Wenda Chu and Chulin Xie and Boxin Wang and Linyi Li and Lang Yin and Arash Nourian and Han Zhao and Bo Li},
  title = {{FOCUS: Fairness via Agent-Awareness for Federated Learning on Heterogeneous Data}},
  booktitle = {Federated Learning @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=XJhL1XlefX},
}
Figure