HERL: Tiered Federated Learning with Adaptive Homomorphic Encryption using Reinforcement Learning

Jiaxang Tang, Zeshan Fayyaz, Mohammad A. Salahuddin, Raouf Boutaba, Zhi-Li Zhang, Ali AnwarOpenReview 2024 · OpenReview

TL;DR

This paper introduces HERL, a novel approach for Federated Learning (FL) that uses Reinforcement Learning (RL) to dynamically adjust Homomorphic Encryption (HE) parameters. By profiling and tiering clients based on their resources and security needs, HERL optimizes HE settings for each tier, significantly reducing computational overhead and convergence time while maintaining high model accuracy and security.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple clients to collaboratively train a machine learning model without sharing their raw data, preserving privacy. However, to further enhance privacy, Homomorphic Encryption (HE) can be used to encrypt model updates before they are sent to the central server. While HE offers strong privacy guarantees, it introduces substantial computational and communication overheads. This problem is exacerbated in real-world FL scenarios where clients are highly heterogeneous, possessing varying computational power, network bandwidth, and different security requirements. Existing FL systems often use a uniform HE configuration for all clients, which is inefficient and impractical for such diverse environments. This paper aims to bridge this gap by proposing an adaptive HE parameter selection mechanism tailored to client heterogeneity.

Methods

The core of HERL is a Reinforcement Learning (RL) agent, specifically using Q-learning, to dynamically optimize HE parameters. The process involves several key steps:

  1. Client Profiling and Tiering: Clients are first profiled to gather information about their computational resources (CPU, GPU, memory, network) and security requirements. Based on this profiling, clients are clustered into tiers. The paper uses hierarchical clustering (FL+HC) as an example, tiering clients based on security requirements and latency. This creates groups of clients with similar characteristics.
  2. RL-based Parameter Selection: For each client tier, an RL agent is employed to select the optimal HE parameter plan. This plan consists of the polynomial modulus degree ($N$) and the coefficient modulus ($q$). The RL agent learns to balance the trade-off between HE latency, model utility (accuracy), and security.
  3. Q-Learning Mechanism: The RL agent uses a Q-table to store state-action values. A "state" represents the characteristics of a client tier, and an "action" is selecting a specific $(q, N)$ pair. The agent learns through exploration (random actions) and exploitation (choosing the best-known action) to maximize a reward function.
  4. Reward Function: A custom reward function is designed to balance utility gains, minimize round time, and penalize security loss. This function guides the RL agent towards optimal parameter choices that satisfy both performance and security objectives.

Datasets

The experiments were conducted using the following datasets:

  • CIFAR-10: A widely-used dataset for image classification.
  • CIFAR-100: A more challenging version of CIFAR-10 with a larger number of classes.

The paper simulates a realistic FL environment by partitioning these datasets in a non-IID (non-Independent and Identically Distributed) manner using a Dirichlet distribution, mimicking real-world data variability across clients.

Key Insights and Ideas

  • Heterogeneity is Key: The paper strongly emphasizes that a one-size-fits-all approach to HE parameters in FL is inefficient due to client heterogeneity. Adaptive strategies are crucial.
  • RL for Dynamic Optimization: Using RL (specifically Q-learning) is a novel way to dynamically adjust HE parameters in real-time, adapting to changing client conditions and requirements.
  • Tiering for Granularity: Clustering clients into tiers based on their characteristics allows for more granular and effective optimization of HE parameters, rather than a single global setting.
  • Balancing Act: The core challenge is balancing computational efficiency (latency, overhead) with privacy (security) and model performance (utility). HERL's reward function and RL agent are designed to achieve this balance.
  • Trade-offs are Inevitable: The paper clearly illustrates that higher security often comes with higher latency and computational cost, and vice-versa. HERL aims to find the "sweet spot" for each client tier.

Key Results

HERL demonstrated significant improvements over baseline approaches (including uniform HE and heuristic methods) in comprehensive experiments:

  • Improved Utility: HERL achieved up to a 17% improvement in final accuracy on the CIFAR-100 dataset with IFCA clustering.
  • Reduced Convergence Time: Convergence time was reduced by at least 15% across all scenarios, with a maximum reduction of up to 24% on CIFAR-10 with TiFL clustering.
  • Increased Convergence Efficiency: HERL showed up to a 30% increase in convergence efficiency on CIFAR-100, meaning it achieved better accuracy faster.
  • Adaptive Parameter Selection: Figure 6 shows that HERL's selected HE parameter distributions are more varied and tailored to different tiers compared to the uniform baseline or the heuristic approach.

Limitations and Future Work

  • Threat Model: The current work focuses on a semi-honest (honest-but-curious) threat model. It does not consider malicious clients who might actively try to poison the model.
  • Clustering Approach: While HERL is designed to be generic, the experiments primarily use FL+HC for tiering. Exploring other clustering methods could be beneficial.
  • Optimal Number of Clusters: The paper notes that while increasing the number of clusters generally improves performance, there's an optimal point beyond which overhead increases, leading to a slight decline. Further research into determining this optimal number is suggested.
  • Future Work: The authors suggest exploring further optimization techniques and extending HERL to more complex and decentralized FL environments. This could involve investigating more advanced RL algorithms or different HE schemes.

Summary updated

Abstract

Federated Learning is a well-researched approach for collaboratively training machine learning models across decentralized data while preserving privacy. However, integrating Homomorphic Encryption to ensure data confidentiality introduces significant computational and communication overheads, particularly in heterogeneous environments where clients have varying computational capacities and security needs. In this paper, we propose HERL, a Reinforcement Learning-based approach that uses Q-Learning to dynamically optimize encryption parameters, specifically the polynomial modulus degree, $N$, and the coefficient modulus, $q$, across different client tiers. Our proposed method involves first profiling and tiering clients according to the chosen clustering approach, followed by dynamically selecting the most suitable encryption parameters using an RL-agent. Experimental results demonstrate that our approach significantly reduces the computational overhead while maintaining utility and a high level of security. Empirical results show that HERL improves utility by 17%, reduces the convergence time by up to 24%, and increases convergence efficiency by up to 30%, with minimal security loss.
BibTeX
@misc{tang2024herl,
  author = {Jiaxang Tang and Zeshan Fayyaz and Mohammad A. Salahuddin and Raouf Boutaba and Zhi-Li Zhang and Ali Anwar},
  title = {{HERL: Tiered Federated Learning with Adaptive Homomorphic Encryption using Reinforcement Learning}},
  year = {2024},
  eprint = {2409.07631},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.07631},
}
Figure