Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions

Shivam Gupta, Shivam Gupta 0004, Tarushi, Tsering Wangzes, Shweta JainarXiv 2024 · DOI · arXiv

Abstract

Federated learning has emerged as a powerful paradigm for privacy-preserving machine learning, yet its application to unsupervised tasks like clustering remains challenging due to data heterogeneity. Standard federated clustering methods often produce a single global model that fails to represent diverse local data distributions, leading to performance inequities among participants. We present p-FClus, a novel personalized federated clustering algorithm. By enabling clients to fine-tune global cluster centers to their local data in a single communication round, p-FClus significantly reduces individual clustering costs and ensures fairness. Experimental results across synthetic and real-world datasets demonstrate that p-FClus consistently outperforms state-of-the-art methods in terms of mean cost, cost stability, and worst-case performance.

Introduction

The proliferation of edge devices has led to an explosion of unlabeled data. While Federated Learning (FL) offers a path to utilize this data without compromising privacy, the majority of FL research focuses on supervised learning. Federated Unsupervised Learning (FedUL), particularly clustering, faces unique hurdles. The primary challenge is data heterogeneity: local data distributions often vary significantly across clients. For instance, in a clustering task with nine true underlying distributions, one client might only possess data from three, while another possesses all nine.

Existing State-of-the-Art (SOTA) federated clustering methods, such as k-FED or MFC, typically aim for a single global consensus. However, a "one-size-fits-all" model often results in high "clustering costs" for clients whose local data deviates from the global average. This lack of fairness can discourage participation in decentralized systems. Furthermore, clustering lacks the inherent synchronization found in supervised learning (e.g., fixed weight ordering), making the direct extension of personalization techniques non-trivial.

Methodology: The p-FClus Algorithm

To address these challenges, we propose p-FClus (personalized Federated Clustering). The algorithm introduces a personalization step that allows clients to adapt global knowledge to their specific local contexts. The process is structured into three phases:

  1. Phase 1: Client Initialization: Each client independently executes a local clustering algorithm (e.g., $k$-means or $k$-medoids) on its private data to obtain a set of local cluster centers.
  2. Phase 2: Server Aggregation: Clients transmit these local centers to a central server. The server then performs a "smart procedure" to find the best global centers that capture the union of all true distributions represented across the client pool. These global centers are then broadcast back to all clients.
  3. Phase 3: Client Personalization: This is the core innovation of p-FClus. Instead of adopting the global centers directly, each client fine-tunes them using a mathematically supported personalization procedure. This involves a novel objective function that balances minimizing local clustering cost with a regularization term that prevents the personalized centers from straying too far from the global centers.
Overview of the p-FClus framework, contrasting traditional and federated clustering while detailing the proposed three-phase personalized approach.

Notably, p-FClus achieves this personalization in just one communication round, making it highly efficient for resource-constrained edge environments. It is also compatible with any finite $\ell$-norm, supporting various clustering objectives.

Experimental Evaluation

The efficacy of p-FClus was validated against a diverse set of benchmarks, including synthetic datasets with varying cluster overlap and extrinsic real-world datasets such as Adult, Bank, Diabetes, and FMNIST. The researchers also utilized intrinsic federated datasets like FEMNIST (500 writers) and WISDM (36 clients using sensor data) to test performance in naturally decentralized settings.

Key Results

Experimental results consistently show that p-FClus provides superior performance and fairness compared to SOTA baselines:

  • Reduced Clustering Cost: p-FClus achieves a lower mean cost per data point. On the WISDM dataset, the improvement was as high as five orders of magnitude compared to k-FED and MFC.
  • Enhanced Fairness: The "cost deviation" ($\sigma$)—representing the performance gap between clients—is significantly lower and remains stable even as data heterogeneity increases.
  • Minimized Worst-Case Risk: The maximum cost suffered by any individual client is minimized, ensuring that no participant is unfairly penalized by the global model.

Discussion and Future Directions

The success of p-FClus highlights that personalization is a critical tool for achieving fairness in federated unsupervised learning. By bypassing the synchronization problem through point-wise updates and allowing local adaptation, p-FClus bridges the gap between diverse data distributions.

Future research will focus on enhancing the robustness of p-FClus against malicious clients and strategic data manipulation. Additionally, the development of "federated unlearning" techniques—allowing the removal of specific client contributions without full retraining—remains a vital area for ensuring long-term privacy and compliance in decentralized systems.

Summary updated

Abstract

The rapid growth of data from edge devices has catalyzed the performance of machine learning algorithms. However, the data generated resides at client devices thus there are majorly two challenge faced by traditional machine learning paradigms - centralization of data for training and secondly for most the generated data the class labels are missing and there is very poor incentives to clients to manually label their data owing to high cost and lack of expertise. To overcome these issues, there have been initial attempts to handle unlabelled data in a privacy preserving distributed manner using unsupervised federated data clustering. The goal is partition the data available on clients into $k$ partitions (called clusters) without actual exchange of data. Most of the existing algorithms are highly dependent on data distribution patterns across clients or are computationally expensive. Furthermore, due to presence of skewed nature of data across clients in most of practical scenarios existing models might result in clients suffering high clustering cost making them reluctant to participate in federated process. To this, we are first to introduce the idea of personalization in federated clustering. The goal is achieve balance between achieving lower clustering cost and at same time achieving uniform cost across clients. We propose p-FClus that addresses these goal in a single round of communication between server and clients. We validate the efficacy of p-FClus against variety of federated datasets showcasing it's data independence nature, applicability to any finite $\ell$-norm, while simultaneously achieving lower cost and variance.
BibTeX
@misc{gupta2024fair,
  author = {Shivam Gupta and Shivam Gupta and Tarushi and Tsering Wangzes and Shweta Jain},
  title = {{Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions}},
  year = {2024},
  eprint = {2407.04302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.04302},
}
Figure