A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networks

Nikolaos Koursioumpas, Lina Magoula, Nikolaos Petropouleas, Alexandros-Ioannis Thanopoulo, Theodora Panage, Nancy Alonistiot, M. A. Gutierrez-Esteve, Ramin KhaliliIEEE Transactions on Green Communications and Networking 2023 · DOI · arXiv · OpenReview

TL;DR

This paper tackles the significant energy consumption of AI in wireless networks by proposing a "safe" Deep Reinforcement Learning (DRL) approach for Federated Learning (FL). Their method orchestrates device resources to minimize energy use while ensuring model performance, achieving up to a 94% reduction in total energy consumption compared to existing methods.

What Problem Does This Paper Address?

The increasing integration of Artificial Intelligence (AI) into wireless networks, particularly through Federated Learning (FL), brings substantial environmental concerns due to high energy demands. FL, while preserving privacy by training models locally on devices, still incurs significant energy costs for computation and communication. Existing FL optimization strategies often overlook the environmental impact, and there's a need for methods that can dynamically manage resources in complex, real-world wireless environments while guaranteeing model performance and safety (i.e., adhering to operational constraints).

Methods

The core of the proposed solution is a Soft Actor-Critic (SAC) Deep Reinforcement Learning (DRL) agent. This agent learns to orchestrate computational and communication resources for FL participants. Key innovations include:

  • Safe RL with Penalty Function: A penalty is introduced during the DRL training phase to discourage strategies that violate environmental constraints, ensuring a "safe" learning process.
  • Device-Level Synchronization: Instead of relying on a central coordinator, synchronization happens at the device level. This prevents unnecessary model transmissions from devices that miss time thresholds, reducing communication overhead and energy waste.
  • Computationally Cost-Effective FL Environment: To make the offline training of the SAC agent feasible, a simplified FL environment is created that emulates real FL processes by estimating key parameters like local iterations and global performance. This significantly reduces the computational burden and energy cost of the training phase.

Datasets

  • MNIST: Used for training the FL-CNN model within the simulated FL environment for the offline SAC agent training.
  • CIFAR-10: Used for evaluating the robustness of the trained SAC model on a more complex FL-CNN architecture during the online inference phase.

Key Insights and Ideas

  • "Safety" in RL for FL is Crucial: Simply optimizing for energy efficiency isn't enough; the system must also operate within defined constraints. The penalty function effectively guides the SAC agent towards compliant and thus more reliable solutions.
  • Decentralized Synchronization is More Efficient: Shifting synchronization responsibility to individual devices, rather than a central coordinator, directly cuts down on wasted communication efforts and associated energy.
  • Simulated Environments are Key for Training Complex RL Agents: Training sophisticated RL agents for real-world FL scenarios is prohibitively expensive. A well-designed simulation can capture the essential dynamics without the massive overhead.
  • A Single Trained Agent Can Generalize: The paper demonstrates that a SAC agent trained on one FL architecture (MNIST CNN) can perform well on a different, more complex one (CIFAR-10 CNN) without extensive retraining, highlighting its robustness and generalization capabilities.

Key Results

  • Massive Energy Reduction: The proposed SAC-based approach achieved up to a 94% decrease in total energy consumption compared to four state-of-the-art baseline schedulers (Best Effort, Random Selection, Greedy Selection, and Genetic Algorithm Scheduler).
  • Reduced Constraint Violations: The "safe" RL approach significantly minimized constraint violations, averaging around 0.85 violations per worker across different scenarios.
  • Scalability: The energy consumption per worker remained relatively stable even as the number of workers increased from 5 to 20.
  • Robustness: The solution demonstrated strong performance across static and dynamic network environments and different FL architectures (MNIST and CIFAR-10 CNNs).
  • Trade-off: While energy efficient, the SAC approach showed a slower convergence speed compared to some faster but less energy-efficient baselines.

Limitations and Future Work

  • Offline Training Complexity: Although a simulated environment was used, the offline training phase can still be computationally intensive and require millions of episodes for convergence, especially with more workers.
  • Future Work: The authors suggest investigating energy-efficient FL approaches that incorporate renewable energy sources to further improve the carbon footprint of FL in wireless networks. They also propose examining additional resource types beyond computation and communication.

Summary updated

Abstract

Progressing towards a new era of Artificial Intelligence (AI) -enabled wireless networks, concerns regarding the environmental impact of AI have been raised both in industry and academia. Federated Learning (FL) has emerged as a key privacy preserving decentralized AI technique. Despite efforts currently being made in FL, its environmental impact is still an open problem. Targeting the minimization of the overall energy consumption of an FL process, we propose the orchestration of computational and communication resources of the involved devices to minimize the total energy required, while guaranteeing a certain performance of the model. To this end, we propose a Soft Actor Critic Deep Reinforcement Learning (DRL) solution, where a penalty function is introduced during training, penalizing the strategies that violate the constraints of the environment, and contributing towards a safe RL process. A device level synchronization method, along with a computationally cost effective FL environment are proposed, with the goal of further reducing the energy consumption and communication overhead. Evaluation results show the effectiveness and robustness of the proposed scheme compared to four state-of-the-art baseline solutions on different network environments and FL architectures, achieving a decrease of up to 94% in the total energy consumption.
BibTeX
@misc{koursioumpas2023safe,
  author = {Nikolaos Koursioumpas and Lina Magoula and Nikolaos Petropouleas and Alexandros-Ioannis Thanopoulo and Theodora Panage and Nancy Alonistiot and M. A. Gutierrez-Esteve and Ramin Khalili},
  title = {{A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networks}},
  year = {2023},
  eprint = {2308.10664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.10664},
}
Figure