Personalized Federated Learning Algorithm with Adaptive Clustering for Non-IID IoT Data Incorporating Multi-Task Learning and Neural Network Model Characteristics

Hua-Yang Hsu, Kay Hooi Keoy, Junru Chen, Han‐Chieh Chao, Chin‐Feng LaiSensors 2023 · DOI

TL;DR

This paper introduces FedAC, a novel personalized federated learning algorithm that uses adaptive clustering to address the challenges of non-identically distributed (Non-IID) data in IoT environments. By dynamically grouping clients with similar data distributions, FedAC significantly improves model accuracy and efficiency, offering a more robust and privacy-conscious solution for real-world IoT applications.

What Problem Does This Paper Address?

Federated learning (FL) has emerged as a powerful paradigm for training machine learning models on decentralized data while preserving privacy. However, practical FL deployments, especially in Internet of Things (IoT) settings, face significant hurdles. One of the most critical is data heterogeneity, where data across different devices (clients) is not Independent and Identically Distributed (Non-IID). This Non-IID nature can severely degrade model performance, leading to inaccurate predictions and slow convergence. Existing FL methods often struggle with this, as they typically assume IID data or rely on simple aggregation strategies that can be biased by skewed client data. Furthermore, the inherent communication costs and device heterogeneity in IoT environments add further complexity. This paper aims to tackle these challenges by developing an FL algorithm that can effectively handle Non-IID data through intelligent client grouping.

Methods

The core innovation of this paper is the Federated Adaptive Cluster (FedAC) algorithm. Unlike traditional clustering methods that require pre-defined cluster numbers, FedAC employs an automatic, adaptive clustering approach. Here's a breakdown of the key components:

  • Personalized Federated Learning Framework: FedAC builds upon the standard federated learning structure, involving a central server and multiple client devices.
  • Adaptive Clustering: The algorithm introduces a novel clustering mechanism that is triggered after a substantial amount of training has occurred. This timing is crucial to ensure that the model parameters have reached a state suitable for accurate clustering.
  • Clustering Criteria: The decision to cluster is based on two main conditions:
  • Loss Reduction: Assessing if the global model is nearing convergence.
  • Gradient Similarity: Using cosine similarity to compare client gradient updates with the global model's gradient update. Significant deviations suggest differing data distributions.
  • Dichotomy Approach: When clustering is initiated, clients are divided into two groups using a dichotomy approach, aiming to separate clients with distinct data distributions. This process is designed to be independent of specific distance thresholds or the global model itself.
  • Multi-Task Learning and Neural Network Characteristics: While the abstract mentions these, the detailed methodology section focuses more on the clustering aspect. The paper implies that the algorithm leverages these to enhance the learning process within the federated setting.
  • Federated Averaging (FedAvg) Integration: For aggregation and local training, FedAC utilizes the well-established FedAvg algorithm as a baseline, highlighting its improvements over standard FedAvg in Non-IID scenarios.

Datasets

The experiments were conducted using the CIFAR-10 dataset. This dataset is generally considered IID. To simulate realistic Non-IID scenarios, the authors artificially created four distinct Non-IID data distributions across clients:

  1. IID Situation: Clients have data from all classes with equal probability.
  2. Non-IID Situation (two clusters): Clients are divided into two groups, each with five classes.
  3. Non-IID Situation (three clusters): Clients are divided into three groups, with two groups having three classes and one group having four classes.
  4. Non-IID Situation (four clusters): Clients are divided into four groups, with two groups having two classes and two groups having three classes.

These scenarios were designed to evaluate the algorithm's performance across varying degrees of data distribution heterogeneity.

Key Insights and Ideas

  • Clustering After Training is Key: The decision to delay clustering until after significant training is a smart move. It allows the model to learn initial patterns, making the subsequent clustering more meaningful and less susceptible to initial data noise or imbalances.
  • Gradient Similarity as a Proxy for Data Distribution: Using the cosine similarity between client and global gradient updates as a metric for data distribution similarity is an elegant way to infer client data characteristics without directly accessing the data itself, thus preserving privacy.
  • Automatic Clustering Outperforms Fixed Clusters: The paper's emphasis on automatic clustering, rather than pre-defining the number of clusters, makes the algorithm more adaptable to diverse and unknown data distributions in real-world IoT deployments.
  • FedAC Stabilizes Non-IID Performance: While FedAvg struggles with oscillations in Non-IID scenarios, FedAC demonstrates a notable stabilization of accuracy and loss after its clustering phase.

Key Results

  • Improved Accuracy in Non-IID Scenarios: The experimental results show that FedAC significantly improves model accuracy in Non-IID settings compared to FedAvg. Figures 4, 5, 6, and 7 illustrate this stabilization and higher accuracy for FedAC, especially as the number of clusters increases.
  • Comparable Performance in IID Scenarios: In IID situations, FedAC's performance is nearly identical to FedAvg, indicating that its clustering mechanism doesn't introduce unnecessary overhead when data is already well-distributed.
  • Trade-off Between Clustering Frequency and Resource Consumption: The paper notes that as the number of client clusters increases, the frequency of clustering rises, leading to higher computational resource consumption. This is identified as an area for future optimization.

Limitations and Future Work

  • Computational Cost of Clustering: The authors acknowledge that scanning all clients within a cluster during each clustering round can be computationally demanding. Future work should focus on optimizing this process to reduce computation time and address issues with straggling clients.
  • Clustering Initiation Criteria: The current threshold-based criteria for initiating clustering are described as "straightforward." Future research should explore more sophisticated and adaptable detection methods for optimal clustering timing, especially in scenarios with oscillating loss.
  • Alternative Aggregation Strategies: While FedAC uses FedAvg, the authors suggest exploring alternative global strategies like Federated Proximal (FedProx) to potentially further enhance performance.
  • Privacy Concerns with Parameter-Based Clustering: The paper briefly mentions that using model parameters for clustering might raise privacy concerns. Future research should investigate techniques to ensure effective clustering while preserving privacy, such as differential privacy.
  • Optimizing Communication Overhead: While the paper focuses on accuracy, it notes that lossy compression techniques used for communication might impact efficiency. Future work could explore methods to minimize communication overhead without compromising data integrity.

Summary updated

Abstract

The proliferation of IoT devices has led to an unprecedented integration of machine learning techniques, raising concerns about data privacy. To address these concerns, federated learning has been introduced. However, practical implementations face challenges, including communication costs, data and device heterogeneity, and privacy security. This paper proposes an innovative approach within the context of federated learning, introducing a personalized joint learning algorithm for Non-IID IoT data. This algorithm incorporates multi-task learning principles and leverages neural network model characteristics. To overcome data heterogeneity, we present a novel clustering algorithm designed specifically for federated learning. Unlike conventional methods that require a predetermined number of clusters, our approach utilizes automatic clustering, eliminating the need for fixed cluster specifications. Extensive experimentation demonstrates the exceptional performance of the proposed algorithm, particularly in scenarios with specific client distributions. By significantly improving the accuracy of trained models, our approach not only addresses data heterogeneity but also strengthens privacy preservation in federated learning. In conclusion, we offer a robust solution to the practical challenges of federated learning in IoT environments. By combining personalized joint learning, automatic clustering, and neural network model characteristics, we facilitate more effective and privacy-conscious machine learning in Non-IID IoT data settings.
BibTeX
@inproceedings{hsu2023personalized,
  author = {Hua-Yang Hsu and Kay Hooi Keoy and Junru Chen and Han‐Chieh Chao and Chin‐Feng Lai},
  title = {{Personalized Federated Learning Algorithm with Adaptive Clustering for Non-IID IoT Data Incorporating Multi-Task Learning and Neural Network Model Characteristics}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23229016},
}
Figure